Oct 30 21:58:13.051807 ip-10-0-129-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:58:13.051821 ip-10-0-129-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:58:13.051831 ip-10-0-129-125 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:58:13.052166 ip-10-0-129-125 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:58:23.231903 ip-10-0-129-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:58:23.231925 ip-10-0-129-125 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b3af563f81f942f886251835a9ff901e --
Oct 30 22:00:25.163904 ip-10-0-129-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:00:25.548459 ip-10-0-129-125 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:00:25.548459 ip-10-0-129-125 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:00:25.548459 ip-10-0-129-125 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:00:25.548459 ip-10-0-129-125 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:00:25.552437 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552353    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.552437 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552437    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.552437 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552441    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552444    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552447    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552450    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552455    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552458    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552460    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552463    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552466    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552468    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552471    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552480    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552483    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552485    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552488    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552491    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552493    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552496    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552499    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.552531 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552503    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552507    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552511    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552514    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552517    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552519    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552522    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552525    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552527    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552530    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552532    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552535    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552538    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552541    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552544    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552547    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552550    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552552    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552555    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552557    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.552982 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552560    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552563    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552566    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552568    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552572    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552574    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552577    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552594    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552598    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552606    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552610    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552613    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552617    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552620    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552623    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552628    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552632    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552636    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.553446 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552639    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552642    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552645    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552647    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552650    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552653    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552656    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552658    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552661    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552663    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552666    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552669    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552672    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552676    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552678    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552681    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552683    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552688    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552690    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552693    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.553907 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552696    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552698    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552701    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552704    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552706    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552709    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552712    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552715    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552718    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552720    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552723    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552726    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552728    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552731    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552734    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552736    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552739    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552741    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552745    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.554411 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552748    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552750    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552753    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552755    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552758    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552760    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.552763    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554604    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554613    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554616    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554620    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554623    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554626    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554629    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554632    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554634    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554637    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554640    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554643    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554645    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.554870 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554648    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554650    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554653    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554656    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554658    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554661    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554663    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554668    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554671    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554673    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554680    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554682    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554685    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554689    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554691    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554694    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554697    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554699    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554703    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.555359 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554705    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554708    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554710    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554713    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554716    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554718    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554721    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554724    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554726    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554728    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554731    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554733    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554736    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554738    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554741    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554744    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554746    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554749    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554751    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554754    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.555855 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554757    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554759    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554762    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554764    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554767    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554769    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554773    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554776    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554778    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554781    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554783    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554786    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554789    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554792    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554794    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554797    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554799    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554802    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554804    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.556323 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554807    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554809    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554812    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554817    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554821    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554824    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554827    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554829    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554832    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554835    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554837    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554840    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554843    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554845    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554848    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554852    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554856    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554859    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.556790 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554862    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554865    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554868    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554870    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554879    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554882    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554885    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554888    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554892    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554895    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554897    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554900    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554902    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554905    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554907    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.554910    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555385    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555389    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555393    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555395    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.557254 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555398    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555401    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555403    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555406    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555409    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555412    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555415    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555417    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555420    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555423    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555426    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555428    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555431    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555434    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555436    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555439    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555441    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555444    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555447    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.557739 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555450    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555453    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555456    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555460    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555463    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555466    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555469    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555472    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555474    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555477    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555479    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555482    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555484    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555487    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555489    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555492    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555495    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555498    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555501    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.558197 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555503    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555506    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555509    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555512    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555515    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555517    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555520    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555522    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555525    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555528    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555531    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555533    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555536    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555539    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555541    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555544    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555547    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555549    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555552    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555554    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.558660 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555556    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555559    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555562    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555564    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555566    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555569    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555571    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555574    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555576    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555592    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555595    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555597    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555600    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555602    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555605    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555609    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555612    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555614    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555617    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555619    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.559154 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555622    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555625    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555627    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555630    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555632    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555635    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555638    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555640    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555650    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555654    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555657    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555660    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555662    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555665    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555667    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555670    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555672    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555675    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555678    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.559637 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555680    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555683    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555685    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555688    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.555690    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555786    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555801    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555810    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555815    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555820    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555824    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555828    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555833    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555836    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555840    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555843    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555846    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555850    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555853    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555856    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555859    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555862    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555866    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555870    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:00:25.560083 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555873    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555876    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555879    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555883    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555887    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555890    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555893    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555897    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555900    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555904    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555907    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555910    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555913    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555918    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555921    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555924    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555927    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555930    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555933    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555938    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555941    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555944    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555948    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555951    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:00:25.560736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555955    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555958    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555961    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555964    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555968    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555971    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555974    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555977    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555980    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555983    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555986    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555990    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555993    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.555997    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556000    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556003    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556006    2577 flags.go:64] FLAG: --help="false"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556009    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556012    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556015    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556019    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556022    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556026    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:00:25.561359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556028    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556031    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556034    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556037    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556040    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556043    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556046    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556049    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556052    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556055    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556058    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556061    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556068    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556071    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556074    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556080    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556083    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556087    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556090    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556093    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556096    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556099    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556102    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556107    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556110    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:00:25.561930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556114    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556117    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556120    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556123    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556127    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556130    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556133    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556136    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556143    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556146    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556149    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556152    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556155    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556158    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556161    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556164    2577 flags.go:64] FLAG: --port="10250"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556167    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556170    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0114d9045ac87c8f8"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556174    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556179    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556182    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556185    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556190    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556193    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556196    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556199    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:00:25.562573 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556204    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556207    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556210    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556214    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556217    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556220    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556223    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556226    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556229    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556232    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556235    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556238    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556242    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556245    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556250    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556253    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556256    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556260    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556263    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556265    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556271    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556274    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556277    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556281    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556284    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:00:25.563253 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556287    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556291    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556294    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556297    2577 flags.go:64] FLAG: --v="2"
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556306    2577 flags.go:64] FLAG: --version="false"
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556310    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556314    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556318    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556468    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556472    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556475    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556478    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556482    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556485    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556488    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556491    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556494    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556496    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556499    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556501    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556504    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.563854 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556508    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556511    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556513    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556516    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556518    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556521    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556524    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556526    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556529    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556531    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556534    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556538    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556540    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556543    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556548    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556551    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556553    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556556    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556559    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.564358 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556562    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556565    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556567    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556570    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556573    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556575    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556594    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556598    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556602    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556606    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556610    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556613    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556618    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556621    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556623    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556626    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556628    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556631    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556634    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556636    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.564851 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556639    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556642    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556644    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556648    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556652    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556656    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556659    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556662    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556664    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556667    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556669    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556672    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556676    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556679    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556681    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556684    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556687    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556689    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556692    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556695    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.565319 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556698    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556700    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556703    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556705    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556709    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556712    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556714    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556716    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556719    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556722    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556725    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556727    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556730    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556732    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556735    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556739    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556742    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556744    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556747    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.565807 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556749    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556752    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556756    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556760    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556762    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556765    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556769    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556772    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556774    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556777    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556779    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556783    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556786    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556788    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.556817    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:00:25.566288 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556853    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556863    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556866    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556871    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556873    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556876    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556878    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556881    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556884    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556886    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556889    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556892    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556895    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556901    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556903    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556906    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556909    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.566654 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556912    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556914    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556917    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556920    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556923    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556925    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556928    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556931    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556933    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556936    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556938    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556941    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556944    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556946    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556950    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556953    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556955    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556958    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556960    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556963    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.567138 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556965    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556968    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556971    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556973    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556976    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556978    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556982    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556985    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556987    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556990    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556992    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556995    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.556998    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557003    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557006    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557009    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557011    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557014    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557017    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557020    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.567621 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557022    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557025    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557027    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557030    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557032    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557035    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557040    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557043    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557045    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557048    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557050    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557053    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557055    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557058    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557060    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557063    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557066    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557068    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557072    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557075    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.568137 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557077    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557080    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557083    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557085    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557088    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557091    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557096    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557098    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557102    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557106    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557109    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557112    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557115    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557118    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557120    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557123    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557125    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557129    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.568701 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557131    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557134    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557136    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557139    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557142    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557144    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.557148    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.563832    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.563846    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.563858    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.563863    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563922    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563927    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563930    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563934    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563937    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563940    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563943    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563946    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.569157 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563949    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563951    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563954    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563957    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563960    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563963    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563965    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563968    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563971    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563974    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563976    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563979    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563982    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563985    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563987    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563990    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563993    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563995    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.563998    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.569623 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564000    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564005    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564009    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564012    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564015    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564018    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564020    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564023    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564025    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564028    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564030    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564033    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564036    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564038    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564041    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564043    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564046    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564048    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564052    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.570078 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564054    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564057    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564060    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564062    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564064    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564067    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564070    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564072    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564075    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564078    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564080    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564083    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564085    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564088    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564091    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564093    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564096    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564099    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564101    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.570529 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564104    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564106    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564109    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564111    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564114    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564117    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564119    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564122    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564124    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564127    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564129    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564132    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564134    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564137    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564140    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564143    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564146    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564150    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564153    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.570987 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564156    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564159    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564161    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564164    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564166    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564168    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564171    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564174    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564177    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564179    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564182    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564185    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564187    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564190    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564192    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564195    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564198    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564201    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564203    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.571460 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564206    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564208    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564211    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.564238    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564377    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564383    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564387    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564389    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564393    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564396    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564398    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564401    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564404    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564406    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564409    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564411    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.572202 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564414    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564416    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564419    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564421    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564424    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564426    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564429    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564431    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564434    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564436    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564439    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564441    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564444    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564446    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564449    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564451    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564454    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564456    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564459    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564461    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.572861 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564464    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564466    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564468    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564471    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564473    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564477    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564480    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564482    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564485    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564488    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564490    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564493    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564495    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564498    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564500    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564503    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564505    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564508    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564510    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564513    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.573352 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564515    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564519    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564522    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564525    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564528    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564530    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564532    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564535    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564538    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564540    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564543    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564545    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564548    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564550    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564552    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564555    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564559    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564562    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564565    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.573864 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564568    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564570    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564573    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564575    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564591    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564594    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564597    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564601    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564604    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564607    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564610    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564612    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564615    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564617    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564620    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564622    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564625    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564628    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564630    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.574380 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564633    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564635    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564638    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564640    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564643    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564646    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564648    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564651    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564655    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564659    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564662    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564664    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564668    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564670    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564673    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.575110 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.564697    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564734    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564739    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564742    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564745    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564748    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564751    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564754    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564757    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564760    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564763    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564765    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564768    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564771    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564773    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564776    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564778    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564780    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564783    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:00:25.575489 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564789    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564791    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564794    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564796    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564799    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564801    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564805    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564808    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564810    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564813    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564816    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564818    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564821    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564824    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564827    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564829    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564832    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564834    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564836    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:00:25.575951 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564839    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564841    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564844    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564846    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564849    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564851    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564854    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564857    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564860    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564863    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564867    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564870    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564873    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564875    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564878    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564880    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564883    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564885    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564888    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:00:25.576429 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564891    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564894    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564896    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564899    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564901    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564904    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564907    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564909    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564912    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564914    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564917    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564919    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564922    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564924    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564926    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564930    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564934    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564937    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564941    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564944    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:00:25.576906 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564946    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564949    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564952    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564954    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564958    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564961    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564963    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564966    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564968    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564971    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564973    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564976    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564979    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564982    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564984    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564987    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564989    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564991    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564995    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.564997    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:00:25.577376 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565000    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565002    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565004    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565008    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565010    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565013    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565015    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565018    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:25.565020    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.565072    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.567696    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.568644    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.568737    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.568765    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.573751    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:00:25.577941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.577605    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:00:25.584034 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.584018    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:00:25.587184 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.587169    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 22:00:25.588025 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.588003    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:00:25.590524 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.590505    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8de244e1-938a-4c1d-9bff-333a7a861980:/dev/nvme0n1p3 e3fa749d-938e-473d-8bd1-cc2c25ceb39f:/dev/nvme0n1p4]
Oct 30 22:00:25.590601 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.590524    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 22:00:25.595650 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.595527    2577 manager.go:217] Machine: {Timestamp:2026-10-30 22:00:25.594191312 +0000 UTC m=+0.314869705 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114026 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec205f69f1c74a1a12eeb5cc45ce1a84 SystemUUID:ec205f69-f1c7-4a1a-12ee-b5cc45ce1a84 BootID:b3af563f-81f9-42f8-8625-1835a9ff901e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d8:e9:68:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d8:e9:68:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:c7:cb:ad:73:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:00:25.595650 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.595650    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:00:25.595776 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.595765    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:00:25.598420 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.598401    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:00:25.598562 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.598423    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-125.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:00:25.598620 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.598564    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:00:25.598620 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.598572    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:00:25.598620 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.598594    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:00:25.599576 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.599565    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:00:25.600285 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.600276    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:00:25.604342 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.604330    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:00:25.604456 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.604447    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:00:25.605093 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.605084    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:00:25.605128 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.605098    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:00:25.605128 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.605109    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:00:25.605128 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.605117    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:00:25.605128 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.605125    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:00:25.605448 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.605436    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:00:25.607475 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.607455    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:00:25.613076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.613059    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:00:25.613845 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.613833    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:00:25.613893 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.613864    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:00:25.614502 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.614490    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:00:25.614532 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.614506    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:00:25.614532 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.614513    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:00:25.614532 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.614520    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:00:25.614532 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.614525    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:00:25.614532 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.614531    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:00:25.615125 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.615116    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:00:25.615125 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.615126    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:00:25.615177 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.615134    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:00:25.615177 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.615140    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:00:25.615177 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.615148    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:00:25.615177 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.615157    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:00:25.616022 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.616014    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:00:25.616061 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.616023    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:00:25.619746 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.619727    2577 server.go:1263] "Started kubelet"
Oct 30 22:00:25.619877 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.619819    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:00:25.619945 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.619886    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:00:25.619945 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.619927    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:00:25.620539 ip-10-0-129-125 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:00:25.622753 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.622728    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:00:25.623075 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.623064    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:00:25.625612 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.625593    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 22:00:25.628629 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.628613    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:00:25.628711 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.628657    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:00:25.628759 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.628719    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-125.us-west-2.compute.internal\" not found"
Oct 30 22:00:25.629227 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629209    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:00:25.629384 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629367    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:00:25.629436 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629392    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:00:25.629519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629505    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:00:25.629519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629517    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:00:25.630096 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629939    2577 factory.go:55] Registering systemd factory
Oct 30 22:00:25.630096 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.629961    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:00:25.630202 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.630155    2577 factory.go:153] Registering CRI-O factory
Oct 30 22:00:25.630202 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.630168    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:00:25.630256 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.630210    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:00:25.630256 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.630225    2577 factory.go:103] Registering Raw factory
Oct 30 22:00:25.630256 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.630237    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:00:25.631079 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.631055    2577 manager.go:319] Starting recovery of all containers
Oct 30 22:00:25.642723 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.642698    2577 manager.go:324] Recovery completed
Oct 30 22:00:25.648053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.648037    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.651216 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.651195    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.651281 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.651235    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.651281 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.651251    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.651961 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.651943    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:00:25.651961 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.651954    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.652048 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.651958    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:00:25.652048 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.652040    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:00:25.652858 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.652840    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.652937 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.652868    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.652937 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.652878    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.653626 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.653614    2577 policy_none.go:50] "Start"
Oct 30 22:00:25.653685 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.653630    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:00:25.653685 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.653640    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:00:25.654854 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.654843    2577 policy_none.go:44] "Start"
Oct 30 22:00:25.687249 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.687207    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:00:25.690941 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.690925    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:00:25.691617 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.691597    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:00:25.691661 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.691621    2577 server.go:87] "Starting device plugin registration server"
Oct 30 22:00:25.691922 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.691911    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:00:25.691964 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.691926    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:00:25.692081 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.692070    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:00:25.692152 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.692143    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:00:25.692197 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.692154    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:00:25.692720 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.692701    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:00:25.692820 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.692747    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-125.us-west-2.compute.internal\" not found"
Oct 30 22:00:25.758333 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.758309    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-125.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 22:00:25.761257 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.758306    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-125.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 22:00:25.761257 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.758313    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-125.us-west-2.compute.internal.18e36da380175ddc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:25.619701212 +0000 UTC m=+0.340379606,LastTimestamp:2026-10-30 22:00:25.619701212 +0000 UTC m=+0.340379606,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-125.us-west-2.compute.internal,}"
Oct 30 22:00:25.781920 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.781884    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:00:25.783107 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.783088    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:00:25.783107 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.783109    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:00:25.783266 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.783137    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:00:25.783266 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.783170    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 22:00:25.793557 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.793533    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.794342 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.794327    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.794432 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.794361    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.794432 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.794376    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.794432 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.794408    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.815713 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.815628    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 22:00:25.815806 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.815667    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-125.us-west-2.compute.internal.18e36da381f84278  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-125.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:25.651217016 +0000 UTC m=+0.371895419,LastTimestamp:2026-10-30 22:00:25.651217016 +0000 UTC m=+0.371895419,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-125.us-west-2.compute.internal,}"
Oct 30 22:00:25.856372 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.856351    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:00:25.856480 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.856408    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6c2s"
Oct 30 22:00:25.883858 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.883826    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"]
Oct 30 22:00:25.883975 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.883892    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.885381 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.885366    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.885461 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.885395    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.885461 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.885407    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.887618 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.887605    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.887775 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.887760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.887810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.887797    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.888784 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.888767    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.888784 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.888777    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.888919 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.888798    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.888919 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.888799    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.888919 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.888808    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.888919 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.888813    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.891109 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.891096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.891158 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.891120    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:00:25.891877 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.891862    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:00:25.891972 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.891891    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:00:25.891972 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.891904    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:00:25.911568 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.911548    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:00:25.912760 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.912747    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:00:25.912969 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.912955    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:00:25.913930 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.913915    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.us-west-2.compute.internal\" not found" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.915875 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.915858    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.us-west-2.compute.internal\" not found" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.916727 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.916713    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.916769 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:25.916733    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-125.us-west-2.compute.internal\": node \"ip-10-0-129-125.us-west-2.compute.internal\" not found"
Oct 30 22:00:25.930406 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.930385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.930496 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.930409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:25.930496 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:25.930437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95227d9216a597ca813ef85310eb7d77-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"95227d9216a597ca813ef85310eb7d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.029570 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.029536    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.030675 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.030656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.030736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.030712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.030736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.030683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.030814 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.030745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95227d9216a597ca813ef85310eb7d77-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"95227d9216a597ca813ef85310eb7d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.030814 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.030752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.030814 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.030772    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95227d9216a597ca813ef85310eb7d77-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"95227d9216a597ca813ef85310eb7d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.138566 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.138481    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:00:26.141996 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.141977    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.196646 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.196613    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:00:26.216729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.216698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:26.219823 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:26.219806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:00:27.773293 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:27.773108    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 22:00:27.773857 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:27.773470    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 22:00:27.773857 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:27.773469    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 22:00:27.773857 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:27.773441    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a56facd814ea6491ea434b7fb55cee90-852268fd564bd9ac.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-125.us-west-2.compute.internal.18e36da381f8a41c\": read tcp 10.0.129.125:33664->3.16.100.64:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-125.us-west-2.compute.internal.18e36da381f8a41c  default   4877 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-125.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:25 +0000 UTC,LastTimestamp:2026-10-30 22:00:25.885399853 +0000 UTC m=+0.606078249,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-125.us-west-2.compute.internal,}"
Oct 30 22:00:27.811354 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:27.811326    2577 apiserver.go:52] "Watching apiserver"
Oct 30 22:00:27.834551 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:27.834523    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 22:00:28.065891 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.065853    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:55:27 +0000 UTC" deadline="2028-05-31 04:13:09.444583889 +0000 UTC"
Oct 30 22:00:28.065891 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.065882    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13878h12m41.378704595s"
Oct 30 22:00:28.099507 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.099483    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:00:28.099838 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.099820    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-tw7xp","openshift-cluster-node-tuning-operator/tuned-b4q4p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-hfs8l","kube-system/global-pull-secret-syncer-rbvwt","kube-system/konnectivity-agent-4j6gf","openshift-dns/node-resolver-jg2sh","openshift-multus/multus-m48h8","openshift-network-diagnostics/network-check-target-lm2gp","openshift-network-operator/iptables-alerter-pbcdv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf","openshift-image-registry/node-ca-gpmjd","openshift-multus/multus-additional-cni-plugins-fp9vl"]
Oct 30 22:00:28.102387 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.102370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.104455 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.104436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.106453 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.106438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.106514 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.106497    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:28.108520 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.108502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.108584 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.108565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:28.110585 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.110573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.112654 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.112639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.114990 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.114976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.117162 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.117143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:28.117250 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.117196    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:28.119588 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.119572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.122556 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.122539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.124607 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.124588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.127483 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.127462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.135856 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.135839    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:00:28.139358 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-config\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.139428 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.139428 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-netns\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.139428 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhgf\" (UniqueName: \"kubernetes.io/projected/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-kube-api-access-qxhgf\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.139428 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-systemd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.139570 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-script-lib\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.139570 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.139570 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-k8s-cni-cncf-io\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.139570 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-etc-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.139570 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-sys\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.139865 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.139865 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.139865 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.139865 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.139865 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-hosts-file\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.139865 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f71a60-c64f-4960-934a-6c75e8d94546-serviceca\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgfj\" (UniqueName: \"kubernetes.io/projected/e7f71a60-c64f-4960-934a-6c75e8d94546-kube-api-access-vwgfj\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cnibin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-socket-dir-parent\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-etc-kubernetes\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.139988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-tmp\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-kubelet-config\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.140044 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-tmp-dir\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cni-binary-copy\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-multus\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-kubernetes\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-conf\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-systemd\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3c9a-7726-406c-bfdf-f36065f02a81-konnectivity-ca\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-node-log\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.140329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-run\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-sys-fs\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-daemon-config\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-ovn\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-system-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-netns\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-netd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysconfig\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-host\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-tuned\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-cnibin\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28a3c9a-7726-406c-bfdf-f36065f02a81-agent-certs\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-log-socket\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-socket-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-device-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-bin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.140729 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-multus-certs\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7z47\" (UniqueName: \"kubernetes.io/projected/cd533e21-2bdf-4973-99f5-568d4a5b0922-kube-api-access-x7z47\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-kubelet\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovn-node-metrics-cert\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-lib-modules\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f71a60-c64f-4960-934a-6c75e8d94546-host\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4545h\" (UniqueName: \"kubernetes.io/projected/1dc81215-458c-47bb-87e8-ed1893729907-kube-api-access-4545h\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-hostroot\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-var-lib-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxzk\" (UniqueName: \"kubernetes.io/projected/64039614-3fbe-4a3a-b80f-2078ae1cc041-kube-api-access-2kxzk\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzpr\" (UniqueName: \"kubernetes.io/projected/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-kube-api-access-4rzpr\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-registration-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.140991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd533e21-2bdf-4973-99f5-568d4a5b0922-iptables-alerter-script\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-kubelet\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-bin\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-env-overrides\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92d4g\" (UniqueName: \"kubernetes.io/projected/d91ca99d-6276-4875-9d6b-e1e49533a930-kube-api-access-92d4g\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-dbus\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd533e21-2bdf-4973-99f5-568d4a5b0922-host-slash\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-slash\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-var-lib-kubelet\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rms\" (UniqueName: \"kubernetes.io/projected/767b8f6d-306f-45ef-97cb-a5e04e233892-kube-api-access-86rms\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcf9\" (UniqueName: \"kubernetes.io/projected/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-kube-api-access-qhcf9\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-conf-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-systemd-units\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.141804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.141318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-modprobe-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.158155 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158129    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:00:28.158155 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158139    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.158293 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158179    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:00:28.158293 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158196    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.158467 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158451    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75hcr\""
Oct 30 22:00:28.158550 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158533    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:00:28.158593 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158570    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:00:28.158593 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158580    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdfn5\""
Oct 30 22:00:28.158593 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158578    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.158678 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.158590    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.164659 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.164637    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:00:28.164829 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.164813    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:00:28.164904 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.164858    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xd6xx\""
Oct 30 22:00:28.166697 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.166667    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.166778 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.166707    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.166778 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.166743    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfj4m\""
Oct 30 22:00:28.168668 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.168651    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:00:28.168668 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.168659    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:00:28.205144 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.205123    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.205259 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.205161    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.205982 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.205964    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlw52\""
Oct 30 22:00:28.208876 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.208860    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skckn\""
Oct 30 22:00:28.209541 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.209526    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:00:28.209957 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.209931    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.210053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.209967    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:00:28.210053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210024    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6qlsl\""
Oct 30 22:00:28.210053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210040    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.210053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210045    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj92j\""
Oct 30 22:00:28.210053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210030    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzw7r\""
Oct 30 22:00:28.210053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210030    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210049    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210084    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210088    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210101    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210122    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210166    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210179    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:00:28.210311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.210222    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m85dd"
Oct 30 22:00:28.242168 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-env-overrides\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.242267 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.242267 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92d4g\" (UniqueName: \"kubernetes.io/projected/d91ca99d-6276-4875-9d6b-e1e49533a930-kube-api-access-92d4g\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.242267 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-dbus\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.242267 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242248    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.242381 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242269    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-dbus\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.242381 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd533e21-2bdf-4973-99f5-568d4a5b0922-host-slash\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.242381 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-slash\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.242381 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242346    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd533e21-2bdf-4973-99f5-568d4a5b0922-host-slash\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.242381 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-var-lib-kubelet\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.242514 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242384    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-slash\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.242514 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rms\" (UniqueName: \"kubernetes.io/projected/767b8f6d-306f-45ef-97cb-a5e04e233892-kube-api-access-86rms\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.242514 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.242514 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-var-lib-kubelet\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.242623 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.242894 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-env-overrides\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.243204 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.242526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.243398 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.243469 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcf9\" (UniqueName: \"kubernetes.io/projected/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-kube-api-access-qhcf9\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.243469 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-conf-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.243560 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-systemd-units\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.243560 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.243654 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243598    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-conf-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.243654 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-systemd-units\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.243765 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.243715    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-modprobe-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.243877 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.243858    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:28.244022 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.244008    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:28.74392274 +0000 UTC m=+2.259496001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:28.245217 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-modprobe-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.245336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-config\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.245336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.245336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-netns\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.245336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhgf\" (UniqueName: \"kubernetes.io/projected/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-kube-api-access-qxhgf\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.245519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-systemd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.245519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-script-lib\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.245519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.245519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-k8s-cni-cncf-io\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.245519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-etc-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-sys\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-hosts-file\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.245752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f71a60-c64f-4960-934a-6c75e8d94546-serviceca\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgfj\" (UniqueName: \"kubernetes.io/projected/e7f71a60-c64f-4960-934a-6c75e8d94546-kube-api-access-vwgfj\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cnibin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-socket-dir-parent\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-etc-kubernetes\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245930    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-etc-kubernetes\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-hosts-file\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.245973    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-tmp\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cnibin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.246037    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret podName:8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:28.746019257 +0000 UTC m=+2.261592540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret") pod "global-pull-secret-syncer-rbvwt" (UID: "8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-sys\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246058    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246051 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.245908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-etc-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246061    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-socket-dir-parent\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-kubelet-config\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-tmp-dir\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cni-binary-copy\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-multus\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-kubernetes\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246306    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-multus\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-conf\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246359    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-netns\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246370    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246403    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-kubernetes\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246414    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-systemd\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3c9a-7726-406c-bfdf-f36065f02a81-konnectivity-ca\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.246810 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-node-log\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-systemd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246784    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-systemd\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246788    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-node-log\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-kubelet-config\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.246987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-run\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-sys-fs\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-daemon-config\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-ovn\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-system-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247423    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-system-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247443    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-config\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247452    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-run\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247480    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-sys-fs\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-netns\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.249624 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-netd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysconfig\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-conf\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3c9a-7726-406c-bfdf-f36065f02a81-konnectivity-ca\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248050    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f71a60-c64f-4960-934a-6c75e8d94546-serviceca\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.247533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-ovn\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-script-lib\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248214    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-host\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248325    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-k8s-cni-cncf-io\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-tuned\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-cnibin\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28a3c9a-7726-406c-bfdf-f36065f02a81-agent-certs\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-tmp-dir\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-log-socket\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-socket-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-device-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-bin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.250203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-multus-certs\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7z47\" (UniqueName: \"kubernetes.io/projected/cd533e21-2bdf-4973-99f5-568d4a5b0922-kube-api-access-x7z47\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-kubelet\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248763    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cni-binary-copy\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovn-node-metrics-cert\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-lib-modules\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f71a60-c64f-4960-934a-6c75e8d94546-host\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4545h\" (UniqueName: \"kubernetes.io/projected/1dc81215-458c-47bb-87e8-ed1893729907-kube-api-access-4545h\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-hostroot\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-daemon-config\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-var-lib-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.248993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxzk\" (UniqueName: \"kubernetes.io/projected/64039614-3fbe-4a3a-b80f-2078ae1cc041-kube-api-access-2kxzk\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzpr\" (UniqueName: \"kubernetes.io/projected/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-kube-api-access-4rzpr\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-registration-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249058    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-socket-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd533e21-2bdf-4973-99f5-568d4a5b0922-iptables-alerter-script\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.250888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-kubelet\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249096    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-bin\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249215    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-device-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249254    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-bin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-multus-certs\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-kubelet\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249333    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-host\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249417    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-lib-modules\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249457    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f71a60-c64f-4960-934a-6c75e8d94546-host\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-hostroot\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249753    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-registration-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.249807    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-var-lib-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250093    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-kubelet\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250188    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-bin\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250232    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-cnibin\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250561    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd533e21-2bdf-4973-99f5-568d4a5b0922-iptables-alerter-script\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-log-socket\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251343 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250660    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-tmp\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.251839 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250666    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-netns\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251839 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-netd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.251839 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.250736    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysconfig\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.252668 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.252644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-tuned\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.252773 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.252673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28a3c9a-7726-406c-bfdf-f36065f02a81-agent-certs\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.252773 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.252745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovn-node-metrics-cert\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.306494 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.306466    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:00:28.306494 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.306492    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:00:28.306665 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.306505    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cfccn for pod openshift-network-diagnostics/network-check-target-lm2gp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:28.306665 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.306571    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn podName:9efd0f0b-f296-4a99-ad21-c3fa3923bb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:28.806551561 +0000 UTC m=+2.322124821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cfccn" (UniqueName: "kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn") pod "network-check-target-lm2gp" (UID: "9efd0f0b-f296-4a99-ad21-c3fa3923bb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:28.307078 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.307040    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92d4g\" (UniqueName: \"kubernetes.io/projected/d91ca99d-6276-4875-9d6b-e1e49533a930-kube-api-access-92d4g\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.307520 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.307496    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rms\" (UniqueName: \"kubernetes.io/projected/767b8f6d-306f-45ef-97cb-a5e04e233892-kube-api-access-86rms\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.308678 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.308659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcf9\" (UniqueName: \"kubernetes.io/projected/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-kube-api-access-qhcf9\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.308794 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.308749    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgfj\" (UniqueName: \"kubernetes.io/projected/e7f71a60-c64f-4960-934a-6c75e8d94546-kube-api-access-vwgfj\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.309199 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.309172    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhgf\" (UniqueName: \"kubernetes.io/projected/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-kube-api-access-qxhgf\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.312108 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.312084    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzpr\" (UniqueName: \"kubernetes.io/projected/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-kube-api-access-4rzpr\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.312108 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.312089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxzk\" (UniqueName: \"kubernetes.io/projected/64039614-3fbe-4a3a-b80f-2078ae1cc041-kube-api-access-2kxzk\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.312329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.312308    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4545h\" (UniqueName: \"kubernetes.io/projected/1dc81215-458c-47bb-87e8-ed1893729907-kube-api-access-4545h\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.312577 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.312560    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7z47\" (UniqueName: \"kubernetes.io/projected/cd533e21-2bdf-4973-99f5-568d4a5b0922-kube-api-access-x7z47\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.441013 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.440925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:28.447441 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.447404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:00:28.468978 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.468945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:28.487175 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.487147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:00:28.497924 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.497903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m48h8"
Oct 30 22:00:28.506219 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.506200    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:00:28.515359 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.515341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:00:28.535216 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.535190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:00:28.553554 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.553535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:00:28.555493 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:28.555457    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767b8f6d_306f_45ef_97cb_a5e04e233892.slice/crio-97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9 WatchSource:0}: Error finding container 97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9: Status 404 returned error can't find the container with id 97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9
Oct 30 22:00:28.556126 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:28.556108    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6780e67bf7a04b04eabfaeb0ab1bf0.slice/crio-79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b WatchSource:0}: Error finding container 79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b: Status 404 returned error can't find the container with id 79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b
Oct 30 22:00:28.556964 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:28.556942    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95227d9216a597ca813ef85310eb7d77.slice/crio-150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5 WatchSource:0}: Error finding container 150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5: Status 404 returned error can't find the container with id 150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5
Oct 30 22:00:28.560533 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.560519    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:00:28.563442 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:28.563375    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f71a60_c64f_4960_934a_6c75e8d94546.slice/crio-699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3 WatchSource:0}: Error finding container 699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3: Status 404 returned error can't find the container with id 699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3
Oct 30 22:00:28.576103 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.576085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:00:28.584983 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:28.584944    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc81215_458c_47bb_87e8_ed1893729907.slice/crio-35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850 WatchSource:0}: Error finding container 35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850: Status 404 returned error can't find the container with id 35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850
Oct 30 22:00:28.635642 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.635614    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:00:28.752567 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.752538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:28.752792 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.752574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:28.752792 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.752733    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:28.752792 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.752762    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:28.752932 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.752810    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:29.752788705 +0000 UTC m=+3.268361959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:28.752932 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.752828    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret podName:8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:29.75281892 +0000 UTC m=+3.268392162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret") pod "global-pull-secret-syncer-rbvwt" (UID: "8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:28.781673 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.781653    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:00:28.853123 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.853085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:28.853287 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.853265    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:00:28.853287 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.853287    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:00:28.853369 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.853298    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cfccn for pod openshift-network-diagnostics/network-check-target-lm2gp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:28.853369 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:28.853359    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn podName:9efd0f0b-f296-4a99-ad21-c3fa3923bb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:29.853344907 +0000 UTC m=+3.368918148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfccn" (UniqueName: "kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn") pod "network-check-target-lm2gp" (UID: "9efd0f0b-f296-4a99-ad21-c3fa3923bb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:28.992578 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.992517    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerStarted","Data":"35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850"}
Oct 30 22:00:28.993812 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.993789    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gpmjd" event={"ID":"e7f71a60-c64f-4960-934a-6c75e8d94546","Type":"ContainerStarted","Data":"699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3"}
Oct 30 22:00:28.997542 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.995616    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf" event={"ID":"767b8f6d-306f-45ef-97cb-a5e04e233892","Type":"ContainerStarted","Data":"97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9"}
Oct 30 22:00:28.997542 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.996917    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"4b6780e67bf7a04b04eabfaeb0ab1bf0","Type":"ContainerStarted","Data":"79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b"}
Oct 30 22:00:28.998512 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:28.998491    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"95227d9216a597ca813ef85310eb7d77","Type":"ContainerStarted","Data":"150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5"}
Oct 30 22:00:29.212797 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.212740    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:27 +0000 UTC" deadline="2028-08-07 14:10:26.085087096 +0000 UTC"
Oct 30 22:00:29.212797 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.212797    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15520h9m56.872292895s"
Oct 30 22:00:29.390331 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:29.390304    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cafb6fd_d8bd_417e_8b93_d42dbd70d102.slice/crio-a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e WatchSource:0}: Error finding container a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e: Status 404 returned error can't find the container with id a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e
Oct 30 22:00:29.391340 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:29.391312    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64039614_3fbe_4a3a_b80f_2078ae1cc041.slice/crio-83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27 WatchSource:0}: Error finding container 83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27: Status 404 returned error can't find the container with id 83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27
Oct 30 22:00:29.403847 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:29.403825    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8b0fdc_6162_4680_b22b_d1eb1656f35b.slice/crio-f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba WatchSource:0}: Error finding container f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba: Status 404 returned error can't find the container with id f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba
Oct 30 22:00:29.413913 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:29.413886    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28a3c9a_7726_406c_bfdf_f36065f02a81.slice/crio-7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2 WatchSource:0}: Error finding container 7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2: Status 404 returned error can't find the container with id 7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2
Oct 30 22:00:29.448367 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:29.448336    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd824b6ee_c212_4bea_8c42_5dbd4e57ad60.slice/crio-78022027fe6ba5ea304462da259d99fbf91105191534f3c353a6f05c86e10f2e WatchSource:0}: Error finding container 78022027fe6ba5ea304462da259d99fbf91105191534f3c353a6f05c86e10f2e: Status 404 returned error can't find the container with id 78022027fe6ba5ea304462da259d99fbf91105191534f3c353a6f05c86e10f2e
Oct 30 22:00:29.760077 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.760039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:29.760273 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.760092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:29.760273 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.760230    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:29.760392 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.760291    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret podName:8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:31.760274409 +0000 UTC m=+5.275847663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret") pod "global-pull-secret-syncer-rbvwt" (UID: "8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:29.760392 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.760361    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:29.760514 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.760396    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:31.76038393 +0000 UTC m=+5.275957186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:29.861271 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.861230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:29.861910 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.861485    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:00:29.861910 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.861505    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:00:29.861910 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.861518    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cfccn for pod openshift-network-diagnostics/network-check-target-lm2gp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:29.861910 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.861582    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn podName:9efd0f0b-f296-4a99-ad21-c3fa3923bb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:31.861564656 +0000 UTC m=+5.377137927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfccn" (UniqueName: "kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn") pod "network-check-target-lm2gp" (UID: "9efd0f0b-f296-4a99-ad21-c3fa3923bb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:29.989166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.989132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:29.989340 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.989271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:29.989401 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.989360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:29.989453 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.989427    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:29.989505 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:29.989485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:29.989581 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:29.989565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:30.001902 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.001863    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27"}
Oct 30 22:00:30.003648 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.003619    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jg2sh" event={"ID":"4cafb6fd-d8bd-417e-8b93-d42dbd70d102","Type":"ContainerStarted","Data":"a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e"}
Oct 30 22:00:30.007855 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.007827    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b4q4p" event={"ID":"d824b6ee-c212-4bea-8c42-5dbd4e57ad60","Type":"ContainerStarted","Data":"78022027fe6ba5ea304462da259d99fbf91105191534f3c353a6f05c86e10f2e"}
Oct 30 22:00:30.010475 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.010417    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4j6gf" event={"ID":"d28a3c9a-7726-406c-bfdf-f36065f02a81","Type":"ContainerStarted","Data":"7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2"}
Oct 30 22:00:30.012279 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.012255    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m48h8" event={"ID":"4e8b0fdc-6162-4680-b22b-d1eb1656f35b","Type":"ContainerStarted","Data":"f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba"}
Oct 30 22:00:30.014493 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.014470    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pbcdv" event={"ID":"cd533e21-2bdf-4973-99f5-568d4a5b0922","Type":"ContainerStarted","Data":"15ca7cf28021ca7e84c8963fb04888f391883b955c4a76a5dcd0b12a48a5186f"}
Oct 30 22:00:30.213909 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.213867    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:27 +0000 UTC" deadline="2028-07-26 22:28:09.636118292 +0000 UTC"
Oct 30 22:00:30.213909 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:30.213907    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15240h27m39.422216283s"
Oct 30 22:00:31.776376 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:31.776338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:31.777000 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:31.776398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:31.777000 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.776577    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:31.777000 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.776641    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret podName:8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:35.77662318 +0000 UTC m=+9.292196450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret") pod "global-pull-secret-syncer-rbvwt" (UID: "8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:31.777368 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.777350    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:31.777441 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.777411    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:35.777393139 +0000 UTC m=+9.292966383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:31.878374 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:31.878334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:31.878547 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.878502    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:00:31.878547 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.878521    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:00:31.878547 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.878534    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cfccn for pod openshift-network-diagnostics/network-check-target-lm2gp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:31.878716 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.878594    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn podName:9efd0f0b-f296-4a99-ad21-c3fa3923bb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:35.878576529 +0000 UTC m=+9.394149998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfccn" (UniqueName: "kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn") pod "network-check-target-lm2gp" (UID: "9efd0f0b-f296-4a99-ad21-c3fa3923bb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:31.990047 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:31.989270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:31.990047 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.989442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:31.990047 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:31.989855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:31.990047 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.989937    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:31.990047 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:31.989977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:31.990047 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:31.990020    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:33.035790 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.035521    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="d9df2f7d98c19b88b3e6992398ef83c30125210486067270dcc5d9ab468cf79d" exitCode=0
Oct 30 22:00:33.036511 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.035699    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerDied","Data":"d9df2f7d98c19b88b3e6992398ef83c30125210486067270dcc5d9ab468cf79d"}
Oct 30 22:00:33.041184 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.041159    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b6780e67bf7a04b04eabfaeb0ab1bf0" containerID="009d7ec6a6d6abb742c13ef9b2c1dc93153c61dc96e30a3c47f036ed9eaee2ea" exitCode=0
Oct 30 22:00:33.041387 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.041197    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"4b6780e67bf7a04b04eabfaeb0ab1bf0","Type":"ContainerDied","Data":"009d7ec6a6d6abb742c13ef9b2c1dc93153c61dc96e30a3c47f036ed9eaee2ea"}
Oct 30 22:00:33.989409 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.989372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:33.989559 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:33.989498    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:33.989631 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.989560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:33.989703 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:33.989644    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:33.989760 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:33.989710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:33.989820 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:33.989787    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:34.052613 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:34.052580    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"95227d9216a597ca813ef85310eb7d77","Type":"ContainerStarted","Data":"a1cb7153bc26da97abfdb9450902be796da1e5bb61ea5e0a2ff9c1d00f4c1fc0"}
Oct 30 22:00:34.056728 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:34.056632    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gpmjd" event={"ID":"e7f71a60-c64f-4960-934a-6c75e8d94546","Type":"ContainerStarted","Data":"82e6a0f72206ef5781806353633050a9c0d8dc373c1942859c5528a4f9ada745"}
Oct 30 22:00:34.062964 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:34.062899    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"4b6780e67bf7a04b04eabfaeb0ab1bf0","Type":"ContainerStarted","Data":"6997cdf5a157ebe8e29a1827200605210a7f5da237cbdbea9f557b63569c747a"}
Oct 30 22:00:34.173226 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:34.172942    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal" podStartSLOduration=7.172923981 podStartE2EDuration="7.172923981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:34.172226249 +0000 UTC m=+7.687799529" watchObservedRunningTime="2026-10-30 22:00:34.172923981 +0000 UTC m=+7.688497245"
Oct 30 22:00:34.416564 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:34.415518    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" podStartSLOduration=7.415500292 podStartE2EDuration="7.415500292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:34.415195618 +0000 UTC m=+7.930768883" watchObservedRunningTime="2026-10-30 22:00:34.415500292 +0000 UTC m=+7.931073556"
Oct 30 22:00:34.416564 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:34.416195    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gpmjd" podStartSLOduration=2.157599184 podStartE2EDuration="7.416180047s" totalImagesPullingTime="5.258580863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:34.295954117 +0000 UTC m=+7.811527382" watchObservedRunningTime="2026-10-30 22:00:34.416180047 +0000 UTC m=+7.931753313"
Oct 30 22:00:35.068530 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.068306    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf" event={"ID":"767b8f6d-306f-45ef-97cb-a5e04e233892","Type":"ContainerStarted","Data":"d64a2a71cd539b61c0129d98ab9331e5bba47284d27de425421317c4050e1284"}
Oct 30 22:00:35.074967 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.074893    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4j6gf" event={"ID":"d28a3c9a-7726-406c-bfdf-f36065f02a81","Type":"ContainerStarted","Data":"3263cddfd42a9c36d81cea97cc8e4e72792ed061e75cf6d80e6c1c7200093483"}
Oct 30 22:00:35.195094 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.194090    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4j6gf" podStartSLOduration=2.899893298 podStartE2EDuration="8.194069447s" totalImagesPullingTime="5.294176149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:35.193786037 +0000 UTC m=+8.709359302" watchObservedRunningTime="2026-10-30 22:00:35.194069447 +0000 UTC m=+8.709642711"
Oct 30 22:00:35.814607 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.814569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:35.814799 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.814631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:35.814860 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.814814    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:35.814911 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.814879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret podName:8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:43.814862374 +0000 UTC m=+17.330435620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret") pod "global-pull-secret-syncer-rbvwt" (UID: "8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:35.815304 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.815283    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:35.815421 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.815339    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:43.815325179 +0000 UTC m=+17.330898421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:35.915855 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.915766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:35.916017 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.915998    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:00:35.916102 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.916025    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:00:35.916102 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.916038    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cfccn for pod openshift-network-diagnostics/network-check-target-lm2gp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:35.916205 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.916156    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn podName:9efd0f0b-f296-4a99-ad21-c3fa3923bb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:43.916138275 +0000 UTC m=+17.431711534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfccn" (UniqueName: "kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn") pod "network-check-target-lm2gp" (UID: "9efd0f0b-f296-4a99-ad21-c3fa3923bb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:35.988768 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.988735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:35.988927 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.988878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:35.989094 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.989023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:35.989185 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.989167    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:35.989249 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:35.989238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:35.989337 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:35.989317    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:36.971350 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:36.970239    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:00:37.091611 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.091538    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf" event={"ID":"767b8f6d-306f-45ef-97cb-a5e04e233892","Type":"ContainerStarted","Data":"25cba9816e647a3f749483cbcd0c8589718449e535f5ea83c4f9eed20c2d3e29"}
Oct 30 22:00:37.904452 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.903573    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:00:36.970255449Z","UUID":"9213294b-54a4-4f24-877c-d2ea4295301c","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:00:37.905495 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.905478    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:00:37.905604 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.905502    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:00:37.989218 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.989197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:37.989574 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:37.989332    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:37.989574 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.989406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:37.989574 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:37.989482    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:37.989574 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:37.989159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:37.989574 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:37.989562    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:38.096725 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:38.096570    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jg2sh" event={"ID":"4cafb6fd-d8bd-417e-8b93-d42dbd70d102","Type":"ContainerStarted","Data":"cd670b4d47a7c93b3b8d56f436e4091f1bc5db74ecec89cbf87382d64c146223"}
Oct 30 22:00:38.099238 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:38.099170    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerStarted","Data":"137d539c1d66fbda26415423aff3d0ed9a2539d5439718a7f3947bc6449db175"}
Oct 30 22:00:38.102174 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:38.101802    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m48h8" event={"ID":"4e8b0fdc-6162-4680-b22b-d1eb1656f35b","Type":"ContainerStarted","Data":"d590be87f34036e9560a82646577480e79232cd443f2429045a8d1f407056223"}
Oct 30 22:00:38.363488 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:38.363417    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jg2sh" podStartSLOduration=3.003362687 podStartE2EDuration="11.363402046s" totalImagesPullingTime="8.360039359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:38.224405972 +0000 UTC m=+11.739979235" watchObservedRunningTime="2026-10-30 22:00:38.363402046 +0000 UTC m=+11.878975309"
Oct 30 22:00:38.490524 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:38.490469    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-m48h8" podStartSLOduration=3.243313802 podStartE2EDuration="11.490456215s" totalImagesPullingTime="8.247142413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:38.490214034 +0000 UTC m=+12.005787298" watchObservedRunningTime="2026-10-30 22:00:38.490456215 +0000 UTC m=+12.006029478"
Oct 30 22:00:39.108646 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.108604    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf" event={"ID":"767b8f6d-306f-45ef-97cb-a5e04e233892","Type":"ContainerStarted","Data":"5a10768a2b3aebd44c77f33628c5163f716f309d6c04ce42e2421b04f700c8c9"}
Oct 30 22:00:39.112704 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.112658    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="137d539c1d66fbda26415423aff3d0ed9a2539d5439718a7f3947bc6449db175" exitCode=0
Oct 30 22:00:39.113596 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.113572    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerDied","Data":"137d539c1d66fbda26415423aff3d0ed9a2539d5439718a7f3947bc6449db175"}
Oct 30 22:00:39.236033 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.235988    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf" podStartSLOduration=2.7898179130000003 podStartE2EDuration="12.235966234s" totalImagesPullingTime="9.446148321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:39.235610642 +0000 UTC m=+12.751183906" watchObservedRunningTime="2026-10-30 22:00:39.235966234 +0000 UTC m=+12.751539546"
Oct 30 22:00:39.710108 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.710058    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:39.710820 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.710804    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:39.989232 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.988788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:39.989232 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.988788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:39.989232 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:39.988908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:39.989232 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:39.988899    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:39.989232 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:39.989003    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:39.989232 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:39.989075    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:40.116930 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:40.116701    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b4q4p" event={"ID":"d824b6ee-c212-4bea-8c42-5dbd4e57ad60","Type":"ContainerStarted","Data":"1a03dd3482608a627892d89c78d2d818609f32d726aafee9767c54caeabbe3f5"}
Oct 30 22:00:40.118658 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:40.118614    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pbcdv" event={"ID":"cd533e21-2bdf-4973-99f5-568d4a5b0922","Type":"ContainerStarted","Data":"5e1a2696b4e8a8d727752101a4efe9223dbe688b603c1257c94d6402f02a67d8"}
Oct 30 22:00:40.245228 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:40.244342    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b4q4p" podStartSLOduration=2.694514366 podStartE2EDuration="13.244324164s" totalImagesPullingTime="10.549809798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:40.243832333 +0000 UTC m=+13.759405596" watchObservedRunningTime="2026-10-30 22:00:40.244324164 +0000 UTC m=+13.759897429"
Oct 30 22:00:41.989482 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:41.989449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:41.990287 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:41.989535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:41.990287 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:41.989650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:41.990287 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:41.989661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:41.990287 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:41.989774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:41.990287 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:41.989848    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:42.126155 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:42.126121    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="004f030a10f3b8906ae030b67f556c9597d7efed31644c2dd3c255e584bb22ab" exitCode=0
Oct 30 22:00:42.126327 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:42.126179    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerDied","Data":"004f030a10f3b8906ae030b67f556c9597d7efed31644c2dd3c255e584bb22ab"}
Oct 30 22:00:42.359010 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:42.358913    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pbcdv" podStartSLOduration=7.001003616 podStartE2EDuration="15.358900464s" totalImagesPullingTime="8.357896848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:40.36136654 +0000 UTC m=+13.876939806" watchObservedRunningTime="2026-10-30 22:00:42.358900464 +0000 UTC m=+15.874473728"
Oct 30 22:00:43.203975 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.203936    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:43.204589 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.204079    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:00:43.204589 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.204577    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:00:43.881548 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.881521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:43.881667 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.881562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:43.881749 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.881674    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:43.881749 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.881676    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:43.881890 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.881758    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.88174093 +0000 UTC m=+33.397314194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:00:43.881890 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.881781    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret podName:8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.881774459 +0000 UTC m=+33.397347699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret") pod "global-pull-secret-syncer-rbvwt" (UID: "8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:00:43.981880 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.981838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:43.982006 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.981971    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:00:43.982006 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.981985    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:00:43.982006 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.981994    2577 projected.go:196] Error preparing data for projected volume kube-api-access-cfccn for pod openshift-network-diagnostics/network-check-target-lm2gp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:43.982127 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.982041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn podName:9efd0f0b-f296-4a99-ad21-c3fa3923bb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.982029152 +0000 UTC m=+33.497602406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfccn" (UniqueName: "kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn") pod "network-check-target-lm2gp" (UID: "9efd0f0b-f296-4a99-ad21-c3fa3923bb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:00:43.988936 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.988911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:43.988936 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.988930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:43.989037 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:43.988958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:43.989037 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.989026    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:43.989108 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.989067    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:43.989108 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:43.989094    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:44.132085 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:44.132000    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="9183f53f4211f74f250e74028b3fb9035eff80008ac84060eba0af5933533100" exitCode=0
Oct 30 22:00:44.132218 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:44.132080    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerDied","Data":"9183f53f4211f74f250e74028b3fb9035eff80008ac84060eba0af5933533100"}
Oct 30 22:00:45.137597 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.137413    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"82ae7d5ed1767c6278136ebc18360d03909f5c26a46ad6adb6953753b29f6a70"}
Oct 30 22:00:45.138055 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.137647    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"1880e0e0000a143c27511c2fc3704fdfa838c7b04bfcc5a5934c3ace62abfa0b"}
Oct 30 22:00:45.138055 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.137724    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"3e83c6bc10123a6ff8c298a43d6b6d0ed9fb7c495d690295274e8ce0275062f1"}
Oct 30 22:00:45.138055 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.137777    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"ff123cac354b6ad970516eb4f17d7f481fccd247fd34e689b4f669a8b2ef4524"}
Oct 30 22:00:45.138055 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.137820    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"3ea7738693c065d45eaf6f8bf2eb531eb7e6ef469aa3c9fa96382dc0f1a46f2b"}
Oct 30 22:00:45.138055 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.137864    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"751754776d58b288baa2c0da126d3863f7e60b71254e6550d903cfa46c5b949a"}
Oct 30 22:00:45.989615 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.989381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:45.989801 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.989430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:45.989801 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:45.989736    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:45.989801 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:45.989465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:45.989950 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:45.989799    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:45.989950 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:45.989889    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:47.145115 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:47.145076    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"ec3814ddcfb7d710ca02eb57e88064aaf52916e7f1a42dd386aa1daf200939e5"}
Oct 30 22:00:47.147946 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:47.147921    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerStarted","Data":"7b049ea5d933c7c0a2ee2b802bf62585832fd0c62ccfd94ac745223ccab2b0b8"}
Oct 30 22:00:47.988796 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:47.988761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:47.988957 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:47.988833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:47.988957 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:47.988850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:47.988957 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:47.988927    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:47.989080 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:47.989058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:47.989140 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:47.989126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:48.153169 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:48.153133    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="7b049ea5d933c7c0a2ee2b802bf62585832fd0c62ccfd94ac745223ccab2b0b8" exitCode=0
Oct 30 22:00:48.153569 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:48.153187    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerDied","Data":"7b049ea5d933c7c0a2ee2b802bf62585832fd0c62ccfd94ac745223ccab2b0b8"}
Oct 30 22:00:49.158166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:49.158137    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="ffab3aab2d0787d1e399a09bdfc52b2908b6ba8debbdb576a8b459b6ab29c481" exitCode=0
Oct 30 22:00:49.158561 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:49.158197    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerDied","Data":"ffab3aab2d0787d1e399a09bdfc52b2908b6ba8debbdb576a8b459b6ab29c481"}
Oct 30 22:00:49.989341 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:49.989304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:49.989507 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:49.989431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:49.989507 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:49.989456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:49.989507 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:49.989446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:49.989707 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:49.989530    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:49.989707 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:49.989582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:50.164127 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.164093    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" event={"ID":"1dc81215-458c-47bb-87e8-ed1893729907","Type":"ContainerStarted","Data":"4ca42719984d308795f2486263c0c6f4c6f6f0e4078e782ce1e948d2f74b4868"}
Oct 30 22:00:50.167698 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.167659    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" event={"ID":"64039614-3fbe-4a3a-b80f-2078ae1cc041","Type":"ContainerStarted","Data":"52d9eddea6dde290c69f37c4c97e4bbb40df3b8d69e4ad2cd653d058df2be25f"}
Oct 30 22:00:50.168021 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.167988    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:50.168133 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.168067    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:50.183374 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.183224    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:50.183473 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.183453    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:50.293275 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:50.293163    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fp9vl" podStartSLOduration=9.267301381 podStartE2EDuration="23.293149598s" totalImagesPullingTime="14.025848217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:50.292788158 +0000 UTC m=+23.808361422" watchObservedRunningTime="2026-10-30 22:00:50.293149598 +0000 UTC m=+23.808722861"
Oct 30 22:00:51.169481 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.169447    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:00:51.860752 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.860678    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp" podStartSLOduration=9.56887017 podStartE2EDuration="24.860664698s" totalImagesPullingTime="15.291794528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:50.441310759 +0000 UTC m=+23.956884022" watchObservedRunningTime="2026-10-30 22:00:51.860664698 +0000 UTC m=+25.376237958"
Oct 30 22:00:51.861094 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.861073    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbvwt"]
Oct 30 22:00:51.861211 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.861199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:51.861306 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:51.861284    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:51.861860 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.861843    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lm2gp"]
Oct 30 22:00:51.861939 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.861929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:51.862014 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:51.862000    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:51.912068 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.912042    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hfs8l"]
Oct 30 22:00:51.912197 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:51.912153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:51.912255 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:51.912238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:52.171565 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:52.171500    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:00:52.989214 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:52.989178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:52.989359 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:52.989315    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:53.587356 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:53.587282    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:00:53.988550 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:53.988510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:53.988738 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:53.988572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:53.988738 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:53.988643    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:53.988851 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:53.988788    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:54.989486 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:54.989450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:54.990067 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:54.989556    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:55.988470 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:55.988439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:55.988470 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:55.988472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:55.988712 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:55.988552    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hfs8l" podUID="d91ca99d-6276-4875-9d6b-e1e49533a930"
Oct 30 22:00:55.988712 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:55.988698    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm2gp" podUID="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c"
Oct 30 22:00:56.989720 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:56.989674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:56.990147 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:56.989797    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbvwt" podUID="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165"
Oct 30 22:00:57.802074 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:57.802043    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeReady"
Oct 30 22:00:57.802229 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:57.802161    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:00:57.962096 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:57.961814    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-csjz8"]
Oct 30 22:00:58.015768 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.015737    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dn4w"]
Oct 30 22:00:58.016177 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.015883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:00:58.028813 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.028770    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cgm7l"]
Oct 30 22:00:58.030442 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.029719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:58.030442 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.029735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:00:58.030442 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.029888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:00:58.042965 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.042944    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"]
Oct 30 22:00:58.043067 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.043053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:00:58.056844 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.056826    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-g99mq"]
Oct 30 22:00:58.056973 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.056942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.070985 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.070960    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d45ff7cc7-twst9"]
Oct 30 22:00:58.071110 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.071083    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.074149 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.074131    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.074274 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.074196    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.084418 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.084381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jt6r\" (UniqueName: \"kubernetes.io/projected/5240e9dc-63a3-436f-af94-ab88f6af0c36-kube-api-access-5jt6r\") pod \"kas-connection-checker-685895c7c9-csjz8\" (UID: \"5240e9dc-63a3-436f-af94-ab88f6af0c36\") " pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:00:58.085528 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.085510    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.089701 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.089670    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rstdk\""
Oct 30 22:00:58.089795 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.089679    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:00:58.089843 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.089796    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\""
Oct 30 22:00:58.093069 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.093053    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-q2lps"]
Oct 30 22:00:58.093498 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.093474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.113196 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.113174    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:00:58.113338 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.113323    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s28xf\""
Oct 30 22:00:58.121041 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.121022    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-cxs8w"]
Oct 30 22:00:58.121166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.121152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.127811 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.127793    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8d2dc\""
Oct 30 22:00:58.143216 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.143194    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"]
Oct 30 22:00:58.143319 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.143305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.163712 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.163665    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-876dbf8dd-czb2r"]
Oct 30 22:00:58.163826 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.163727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.174702 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.174666    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:00:58.174828 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.174764    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.174828 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.174763    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.174828 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.174786    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:00:58.174828 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.174822    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.175572 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.175555    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.176055 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.176038    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.176230 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.176216    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:00:58.179169 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.179150    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:00:58.184926 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.184903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185006 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.184941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185006 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.184972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185074 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-config\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.185074 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69e320d2-2d3d-45fb-a44b-41bfe19091ae-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.185201 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185233 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185275 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jt6r\" (UniqueName: \"kubernetes.io/projected/5240e9dc-63a3-436f-af94-ab88f6af0c36-kube-api-access-5jt6r\") pod \"kas-connection-checker-685895c7c9-csjz8\" (UID: \"5240e9dc-63a3-436f-af94-ab88f6af0c36\") " pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:00:58.185311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-config\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.185421 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47rc\" (UniqueName: \"kubernetes.io/projected/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-kube-api-access-p47rc\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.185467 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-serving-cert\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.185510 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-trusted-ca\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.185561 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185605 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-serving-cert\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.185642 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgwv\" (UniqueName: \"kubernetes.io/projected/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-kube-api-access-glgwv\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.185673 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2shr\" (UniqueName: \"kubernetes.io/projected/03d17bd7-a7c6-45b9-b0e5-2d9809543781-kube-api-access-n2shr\") pod \"kas-connection-checker-685895c7c9-7dn4w\" (UID: \"03d17bd7-a7c6-45b9-b0e5-2d9809543781\") " pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:00:58.185748 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185748 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxt6q\" (UniqueName: \"kubernetes.io/projected/1a535db2-2694-4965-b4f9-5b478681381e-kube-api-access-lxt6q\") pod \"kas-connection-checker-685895c7c9-cgm7l\" (UID: \"1a535db2-2694-4965-b4f9-5b478681381e\") " pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:00:58.185849 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.185849 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.185796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.186037 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.186020    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"]
Oct 30 22:00:58.186149 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.186134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.198767 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.198749    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86b69957bf-6kffb"]
Oct 30 22:00:58.198870 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.198856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.210622 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.210606    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"]
Oct 30 22:00:58.210737 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.210724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.222935 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.222916    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"]
Oct 30 22:00:58.223034 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.223021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.227452 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227408    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:00:58.227544 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227463    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:00:58.227544 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227470    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:00:58.227544 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227475    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fskfb\""
Oct 30 22:00:58.227544 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227500    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:00:58.227755 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227584    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hpnbl\""
Oct 30 22:00:58.227755 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227705    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:00:58.227881 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.227859    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:00:58.236424 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.236407    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"]
Oct 30 22:00:58.236519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.236510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:00:58.255769 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.255745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jt6r\" (UniqueName: \"kubernetes.io/projected/5240e9dc-63a3-436f-af94-ab88f6af0c36-kube-api-access-5jt6r\") pod \"kas-connection-checker-685895c7c9-csjz8\" (UID: \"5240e9dc-63a3-436f-af94-ab88f6af0c36\") " pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:00:58.255869 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.255782    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s96ng"]
Oct 30 22:00:58.255869 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.255864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:00:58.274370 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.274337    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zzj6z"]
Oct 30 22:00:58.274506 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.274467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.276669 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276652    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.276795 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276742    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.276795 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276750    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.276795 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276747    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:00:58.276909 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276787    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:00:58.276909 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276819    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.276909 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.276823    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.277034 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277020    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.277076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277021    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:00:58.277076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277024    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.277308 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277296    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.277374 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277358    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:00:58.277797 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277782    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:00:58.277873 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277847    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.277873 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.277853    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:00:58.278144 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.278129    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.286716 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.286824 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4n7\" (UniqueName: \"kubernetes.io/projected/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-kube-api-access-zg4n7\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.286824 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be68f52a-8bf4-474e-b28d-133aecbfb229-serving-cert\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.286824 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.286939 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.286939 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.286939 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.286878    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:00:58.286939 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-stats-auth\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.286939 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.286927    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.786909107 +0000 UTC m=+32.302482349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:00:58.286939 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69e320d2-2d3d-45fb-a44b-41bfe19091ae-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.287166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.286961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-tmp\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.287166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-default-certificate\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.287166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-config\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.287166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.287166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f572fd28-f04a-45ed-a153-5921471fb276-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slx6l\" (UniqueName: \"kubernetes.io/projected/be68f52a-8bf4-474e-b28d-133aecbfb229-kube-api-access-slx6l\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-serving-cert\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9wd\" (UniqueName: \"kubernetes.io/projected/56a790e5-683e-4dfd-82e1-1bde4629e09f-kube-api-access-dx9wd\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2shr\" (UniqueName: \"kubernetes.io/projected/03d17bd7-a7c6-45b9-b0e5-2d9809543781-kube-api-access-n2shr\") pod \"kas-connection-checker-685895c7c9-7dn4w\" (UID: \"03d17bd7-a7c6-45b9-b0e5-2d9809543781\") " pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.287400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.287770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxt6q\" (UniqueName: \"kubernetes.io/projected/1a535db2-2694-4965-b4f9-5b478681381e-kube-api-access-lxt6q\") pod \"kas-connection-checker-685895c7c9-cgm7l\" (UID: \"1a535db2-2694-4965-b4f9-5b478681381e\") " pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:00:58.287770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69e320d2-2d3d-45fb-a44b-41bfe19091ae-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.287770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.287770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-snapshots\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.287770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.287770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-config\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.288031 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld84k\" (UniqueName: \"kubernetes.io/projected/a461f5f1-641c-4f5e-8f29-995036f2f0f7-kube-api-access-ld84k\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.288031 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.288031 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.288031 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.287918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.288236 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.288236 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-config\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.288236 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.288433 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.288433 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.288433 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f572fd28-f04a-45ed-a153-5921471fb276-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.288433 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.288630 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.288630 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.288630 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-trusted-ca\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.288630 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.288630 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47rc\" (UniqueName: \"kubernetes.io/projected/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-kube-api-access-p47rc\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-serving-cert\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-service-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-config\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgwv\" (UniqueName: \"kubernetes.io/projected/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-kube-api-access-glgwv\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.288882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvqc\" (UniqueName: \"kubernetes.io/projected/f572fd28-f04a-45ed-a153-5921471fb276-kube-api-access-2qvqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.288966    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.288981    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.289379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-trusted-ca\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.289519 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.289437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.789406858 +0000 UTC m=+32.304980102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:00:58.290376 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290148    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:00:58.290376 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-serving-cert\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.290376 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290288    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:00:58.290558 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290381    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.290558 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290452    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wfrf6\""
Oct 30 22:00:58.290558 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290488    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wr55q\""
Oct 30 22:00:58.290558 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290526    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:00:58.290558 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290542    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:00:58.290558 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290545    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290549    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290634    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290663    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gg6vq\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290664    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290712    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-22kdc\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290714    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290729    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x2z2x\""
Oct 30 22:00:58.290908 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.290769    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cllmb\""
Oct 30 22:00:58.291273 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.291003    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-serving-cert\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.291768 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.291750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.292732 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292682    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dn4w"]
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292747    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-csjz8"]
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292765    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cgm7l"]
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292781    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-g99mq"]
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292796    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"]
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292811    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-q2lps"]
Oct 30 22:00:58.292819 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292823    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292837    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-cxs8w"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292854    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d45ff7cc7-twst9"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292871    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-876dbf8dd-czb2r"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292884    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86b69957bf-6kffb"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292898    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292915    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s96ng"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292937    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292962    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzj6z"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.292985    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"]
Oct 30 22:00:58.293138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.293003    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"]
Oct 30 22:00:58.351199 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.351168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:00:58.366597 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.366565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgwv\" (UniqueName: \"kubernetes.io/projected/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-kube-api-access-glgwv\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.366896 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.366875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2shr\" (UniqueName: \"kubernetes.io/projected/03d17bd7-a7c6-45b9-b0e5-2d9809543781-kube-api-access-n2shr\") pod \"kas-connection-checker-685895c7c9-7dn4w\" (UID: \"03d17bd7-a7c6-45b9-b0e5-2d9809543781\") " pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:00:58.370276 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.370247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.371021 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.370990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47rc\" (UniqueName: \"kubernetes.io/projected/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-kube-api-access-p47rc\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.371021 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.371004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxt6q\" (UniqueName: \"kubernetes.io/projected/1a535db2-2694-4965-b4f9-5b478681381e-kube-api-access-lxt6q\") pod \"kas-connection-checker-685895c7c9-cgm7l\" (UID: \"1a535db2-2694-4965-b4f9-5b478681381e\") " pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:00:58.371572 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.371546    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.374931 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.374916    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:00:58.374998 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.374981    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:00:58.375033 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.374981    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:00:58.389975 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.389958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.390067 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.389986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kqtv\" (UniqueName: \"kubernetes.io/projected/63fef56c-376f-4d60-a2c9-c9b02731c5ed-kube-api-access-6kqtv\") pod \"network-check-source-6f78d59f99-zl5rs\" (UID: \"63fef56c-376f-4d60-a2c9-c9b02731c5ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:00:58.390067 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-service-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.390172 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvqc\" (UniqueName: \"kubernetes.io/projected/f572fd28-f04a-45ed-a153-5921471fb276-kube-api-access-2qvqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.390172 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.390277 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4n7\" (UniqueName: \"kubernetes.io/projected/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-kube-api-access-zg4n7\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.390277 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be68f52a-8bf4-474e-b28d-133aecbfb229-serving-cert\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.390478 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.390478 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.390478 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390449    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-service-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.390478 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390470    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd25c\""
Oct 30 22:00:58.390666 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390489    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:00:58.390666 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390548    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqg6l\""
Oct 30 22:00:58.390666 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390575    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:00:58.390666 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390622    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hgnsb\""
Oct 30 22:00:58.390666 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.390652    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:00:58.390901 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.390731    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.890712421 +0000 UTC m=+32.406285684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:00:58.390901 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-stats-auth\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.390901 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-tmp\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.390901 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.390843    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:00:58.390901 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-default-certificate\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.390901 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.390917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4sd9\" (UniqueName: \"kubernetes.io/projected/10e32bef-cfe9-48eb-84e9-44fdebb43090-kube-api-access-q4sd9\") pod \"volume-data-source-validator-58c9f5f46d-hsdm7\" (UID: \"10e32bef-cfe9-48eb-84e9-44fdebb43090\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.390931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.89091295 +0000 UTC m=+32.406486209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/914a456e-5637-4208-98bd-ee4078634291-cert\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f572fd28-f04a-45ed-a153-5921471fb276-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slx6l\" (UniqueName: \"kubernetes.io/projected/be68f52a-8bf4-474e-b28d-133aecbfb229-kube-api-access-slx6l\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9wd\" (UniqueName: \"kubernetes.io/projected/56a790e5-683e-4dfd-82e1-1bde4629e09f-kube-api-access-dx9wd\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrzb\" (UniqueName: \"kubernetes.io/projected/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-kube-api-access-mbrzb\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391259    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-tmp\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-snapshots\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.391336 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld84k\" (UniqueName: \"kubernetes.io/projected/a461f5f1-641c-4f5e-8f29-995036f2f0f7-kube-api-access-ld84k\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f572fd28-f04a-45ed-a153-5921471fb276-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-tmp-dir\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-config-volume\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.391664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28g6\" (UniqueName: \"kubernetes.io/projected/914a456e-5637-4208-98bd-ee4078634291-kube-api-access-q28g6\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.392076 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.392057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.892039505 +0000 UTC m=+32.407612746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : configmap references non-existent config key: service-ca.crt
Oct 30 22:00:58.392679 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.392260    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f572fd28-f04a-45ed-a153-5921471fb276-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.392679 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.392577    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:00:58.392679 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.392606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-snapshots\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.392679 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.392618    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.89260618 +0000 UTC m=+32.408179438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:00:58.392888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.392742    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.392950 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.392886    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be68f52a-8bf4-474e-b28d-133aecbfb229-serving-cert\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.392950 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.392914    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.393329 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.393304    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.393423 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.393334    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:00:58.393423 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.393351    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:00:58.393423 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.393345    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.393423 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.393396    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.893382832 +0000 UTC m=+32.408956101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:00:58.393790 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.393755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.394261 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.394234    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-default-certificate\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.395198 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.395176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f572fd28-f04a-45ed-a153-5921471fb276-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.395318 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.395270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-stats-auth\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.395318 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.395306    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.417822 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.417789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:00:58.438436 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.438394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:00:58.454973 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.454879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvqc\" (UniqueName: \"kubernetes.io/projected/f572fd28-f04a-45ed-a153-5921471fb276-kube-api-access-2qvqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.456918 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.455974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:00:58.459255 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.457542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slx6l\" (UniqueName: \"kubernetes.io/projected/be68f52a-8bf4-474e-b28d-133aecbfb229-kube-api-access-slx6l\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.459255 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.458184    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4n7\" (UniqueName: \"kubernetes.io/projected/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-kube-api-access-zg4n7\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.459255 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.458673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9wd\" (UniqueName: \"kubernetes.io/projected/56a790e5-683e-4dfd-82e1-1bde4629e09f-kube-api-access-dx9wd\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.462489 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.462042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.463572 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.463547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.467957 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.467505    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld84k\" (UniqueName: \"kubernetes.io/projected/a461f5f1-641c-4f5e-8f29-995036f2f0f7-kube-api-access-ld84k\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.493088 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-config-volume\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.493237 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28g6\" (UniqueName: \"kubernetes.io/projected/914a456e-5637-4208-98bd-ee4078634291-kube-api-access-q28g6\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.493237 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kqtv\" (UniqueName: \"kubernetes.io/projected/63fef56c-376f-4d60-a2c9-c9b02731c5ed-kube-api-access-6kqtv\") pod \"network-check-source-6f78d59f99-zl5rs\" (UID: \"63fef56c-376f-4d60-a2c9-c9b02731c5ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:00:58.493344 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.493344 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4sd9\" (UniqueName: \"kubernetes.io/projected/10e32bef-cfe9-48eb-84e9-44fdebb43090-kube-api-access-q4sd9\") pod \"volume-data-source-validator-58c9f5f46d-hsdm7\" (UID: \"10e32bef-cfe9-48eb-84e9-44fdebb43090\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:00:58.493442 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/914a456e-5637-4208-98bd-ee4078634291-cert\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.493873 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrzb\" (UniqueName: \"kubernetes.io/projected/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-kube-api-access-mbrzb\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.493963 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.493929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-tmp-dir\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.494564 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.494271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-tmp-dir\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.494564 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.494371    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:00:58.494564 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.494423    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:00:58.994407166 +0000 UTC m=+32.509980421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:00:58.494564 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.494541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-config-volume\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.494832 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.494735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:00:58.496548 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.496525    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/914a456e-5637-4208-98bd-ee4078634291-cert\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.507622 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.507596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:00:58.563407 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.562982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4sd9\" (UniqueName: \"kubernetes.io/projected/10e32bef-cfe9-48eb-84e9-44fdebb43090-kube-api-access-q4sd9\") pod \"volume-data-source-validator-58c9f5f46d-hsdm7\" (UID: \"10e32bef-cfe9-48eb-84e9-44fdebb43090\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:00:58.564113 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.563748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrzb\" (UniqueName: \"kubernetes.io/projected/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-kube-api-access-mbrzb\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.564391 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.564359    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kqtv\" (UniqueName: \"kubernetes.io/projected/63fef56c-376f-4d60-a2c9-c9b02731c5ed-kube-api-access-6kqtv\") pod \"network-check-source-6f78d59f99-zl5rs\" (UID: \"63fef56c-376f-4d60-a2c9-c9b02731c5ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:00:58.566589 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.566567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28g6\" (UniqueName: \"kubernetes.io/projected/914a456e-5637-4208-98bd-ee4078634291-kube-api-access-q28g6\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.593274 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.593244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:00:58.631972 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.631939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:00:58.649871 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.649844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:00:58.669504 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.669265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:00:58.782913 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:58.782886    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5240e9dc_63a3_436f_af94_ab88f6af0c36.slice/crio-3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318 WatchSource:0}: Error finding container 3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318: Status 404 returned error can't find the container with id 3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318
Oct 30 22:00:58.784937 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.784911    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-csjz8"]
Oct 30 22:00:58.796735 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.796705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:58.796842 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.796767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:58.796896 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.796850    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:00:58.796896 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.796866    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:00:58.796896 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.796883    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:00:58.797053 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.796917    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.796901649 +0000 UTC m=+33.312474891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:00:58.797053 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.796931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.79692541 +0000 UTC m=+33.312498651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:00:58.827857 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:58.827819    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d17bd7_a7c6_45b9_b0e5_2d9809543781.slice/crio-ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a WatchSource:0}: Error finding container ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a: Status 404 returned error can't find the container with id ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a
Oct 30 22:00:58.829443 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.829420    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dn4w"]
Oct 30 22:00:58.834209 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:58.834180    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e6d4d4_dc9b_44be_8dc1_abc723e7ffe4.slice/crio-a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085 WatchSource:0}: Error finding container a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085: Status 404 returned error can't find the container with id a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085
Oct 30 22:00:58.840419 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.837740    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"]
Oct 30 22:00:58.873303 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.873272    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cgm7l"]
Oct 30 22:00:58.881443 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:58.881416    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a535db2_2694_4965_b4f9_5b478681381e.slice/crio-cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514 WatchSource:0}: Error finding container cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514: Status 404 returned error can't find the container with id cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.898027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.898097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.898212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.898266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.898308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898437    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898453    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898506    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.898488906 +0000 UTC m=+33.414062160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898579    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898616    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.898606047 +0000 UTC m=+33.414179308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898670    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898722    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.898710986 +0000 UTC m=+33.414284242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898788    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.898778356 +0000 UTC m=+33.414351613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : configmap references non-existent config key: service-ca.crt
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898845    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:00:58.898898 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.898875    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.898865722 +0000 UTC m=+33.414438976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:00:58.916724 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:58.916663    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe68f52a_8bf4_474e_b28d_133aecbfb229.slice/crio-52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c WatchSource:0}: Error finding container 52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c: Status 404 returned error can't find the container with id 52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c
Oct 30 22:00:58.918053 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.917942    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-cxs8w"]
Oct 30 22:00:58.972018 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:58.971987    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf572fd28_f04a_45ed_a153_5921471fb276.slice/crio-e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a WatchSource:0}: Error finding container e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a: Status 404 returned error can't find the container with id e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a
Oct 30 22:00:58.974140 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.974118    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"]
Oct 30 22:00:58.988743 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.988718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:58.999207 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:58.999185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:00:58.999353 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.999336    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:00:58.999419 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:58.999402    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:00:59.999384294 +0000 UTC m=+33.514957549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:00:59.015158 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:59.015134    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e32bef_cfe9_48eb_84e9_44fdebb43090.slice/crio-7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af WatchSource:0}: Error finding container 7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af: Status 404 returned error can't find the container with id 7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af
Oct 30 22:00:59.017646 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.017598    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"]
Oct 30 22:00:59.037043 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:59.037017    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d05eba_0c7d_4837_adc6_fa3fee9b4f2b.slice/crio-dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5 WatchSource:0}: Error finding container dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5: Status 404 returned error can't find the container with id dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5
Oct 30 22:00:59.039082 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.039064    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-q2lps"]
Oct 30 22:00:59.039620 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:59.039600    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fef56c_376f_4d60_a2c9_c9b02731c5ed.slice/crio-3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8 WatchSource:0}: Error finding container 3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8: Status 404 returned error can't find the container with id 3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8
Oct 30 22:00:59.041647 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.041626    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"]
Oct 30 22:00:59.044081 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.044063    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\""
Oct 30 22:00:59.044160 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.044080    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:00:59.073565 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:00:59.073539    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914a456e_5637_4208_98bd_ee4078634291.slice/crio-35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217 WatchSource:0}: Error finding container 35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217: Status 404 returned error can't find the container with id 35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217
Oct 30 22:00:59.075227 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.075209    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s96ng"]
Oct 30 22:00:59.186481 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.186447    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs" event={"ID":"63fef56c-376f-4d60-a2c9-c9b02731c5ed","Type":"ContainerStarted","Data":"3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8"}
Oct 30 22:00:59.187603 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.187579    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7" event={"ID":"10e32bef-cfe9-48eb-84e9-44fdebb43090","Type":"ContainerStarted","Data":"7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af"}
Oct 30 22:00:59.188670 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.188651    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh" event={"ID":"f572fd28-f04a-45ed-a153-5921471fb276","Type":"ContainerStarted","Data":"e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a"}
Oct 30 22:00:59.189804 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.189782    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s96ng" event={"ID":"914a456e-5637-4208-98bd-ee4078634291","Type":"ContainerStarted","Data":"35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217"}
Oct 30 22:00:59.191166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.191143    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cgm7l" event={"ID":"1a535db2-2694-4965-b4f9-5b478681381e","Type":"ContainerStarted","Data":"1a022e3d5249c1ed9b154fd456ca5ace7a9c701db15b2165b1c6ebb52859219f"}
Oct 30 22:00:59.191243 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.191175    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cgm7l" event={"ID":"1a535db2-2694-4965-b4f9-5b478681381e","Type":"ContainerStarted","Data":"cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514"}
Oct 30 22:00:59.192258 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.192242    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w" event={"ID":"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4","Type":"ContainerStarted","Data":"a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085"}
Oct 30 22:00:59.193663 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.193645    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dn4w" event={"ID":"03d17bd7-a7c6-45b9-b0e5-2d9809543781","Type":"ContainerStarted","Data":"8ca5d30195edf2aee57006d4f98bee40a12ba1bbc71408c04da874c30efc5e4c"}
Oct 30 22:00:59.193742 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.193674    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dn4w" event={"ID":"03d17bd7-a7c6-45b9-b0e5-2d9809543781","Type":"ContainerStarted","Data":"ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a"}
Oct 30 22:00:59.195073 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.195050    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-csjz8" event={"ID":"5240e9dc-63a3-436f-af94-ab88f6af0c36","Type":"ContainerStarted","Data":"eeb4fa47571327074f5f2600900619326886cd40cc928f3aa1a337d941828e54"}
Oct 30 22:00:59.195136 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.195088    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-csjz8" event={"ID":"5240e9dc-63a3-436f-af94-ab88f6af0c36","Type":"ContainerStarted","Data":"3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318"}
Oct 30 22:00:59.196090 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.196070    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cxs8w" event={"ID":"be68f52a-8bf4-474e-b28d-133aecbfb229","Type":"ContainerStarted","Data":"52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c"}
Oct 30 22:00:59.197070 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.197051    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" event={"ID":"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b","Type":"ContainerStarted","Data":"dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5"}
Oct 30 22:00:59.652874 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.652841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:59.806498 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.806465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:00:59.806702 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.806529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:00:59.806702 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.806623    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:00:59.806702 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.806648    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:00:59.806702 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.806668    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:00:59.806849 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.806728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.806709054 +0000 UTC m=+35.322282310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:00:59.806849 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.806745    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.806736958 +0000 UTC m=+35.322310198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:00:59.854824 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.854789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:59.907220 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:00:59.907220 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:00:59.907427 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:00:59.907427 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:59.907427 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907300    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:00:59.907427 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:59.907427 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.907400427 +0000 UTC m=+35.422973670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:00:59.907427 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907312    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907355    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907480    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:31.907464033 +0000 UTC m=+65.423037283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : secret "metrics-daemon-secret" not found
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907507    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.907500366 +0000 UTC m=+35.423073608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : configmap references non-existent config key: service-ca.crt
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907510    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.907515876 +0000 UTC m=+35.423089117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907545    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.907534823 +0000 UTC m=+35.423108066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:00:59.907644 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.907577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:00:59.907929 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907709    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:00:59.907929 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907724    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:00:59.907929 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:00:59.907756    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:01.907747836 +0000 UTC m=+35.423321077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:00:59.909799 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.909778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:00:59.917845 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:00:59.917822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:00.012364 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.012329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:00.012497 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.012409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:00.013015 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:00.012993    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:00.013098 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:00.013065    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:02.013045586 +0000 UTC m=+35.528618833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:00.020539 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.020508    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:00.087981 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.087944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:00.199825 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.199520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:00.233586 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.233527    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-csjz8" podStartSLOduration=438.233507619 podStartE2EDuration="7m18.233507619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:00.231867606 +0000 UTC m=+33.747440871" watchObservedRunningTime="2026-10-30 22:01:00.233507619 +0000 UTC m=+33.749080882"
Oct 30 22:01:00.337018 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:01:00.330092    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebd318a_c0d2_45e5_8eeb_f5d85fa0b165.slice/crio-7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36 WatchSource:0}: Error finding container 7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36: Status 404 returned error can't find the container with id 7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36
Oct 30 22:01:00.337018 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.331036    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbvwt"]
Oct 30 22:01:00.471525 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.471440    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7dn4w" podStartSLOduration=438.471419932 podStartE2EDuration="7m18.471419932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:00.470357413 +0000 UTC m=+33.985930677" watchObservedRunningTime="2026-10-30 22:01:00.471419932 +0000 UTC m=+33.986993187"
Oct 30 22:01:00.472287 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.472240    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cgm7l" podStartSLOduration=438.472228477 podStartE2EDuration="7m18.472228477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:00.352336131 +0000 UTC m=+33.867909396" watchObservedRunningTime="2026-10-30 22:01:00.472228477 +0000 UTC m=+33.987801742"
Oct 30 22:01:00.602888 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:00.602852    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lm2gp"]
Oct 30 22:01:00.616891 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:01:00.616855    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efd0f0b_f296_4a99_ad21_c3fa3923bb7c.slice/crio-d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3 WatchSource:0}: Error finding container d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3: Status 404 returned error can't find the container with id d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3
Oct 30 22:01:01.138258 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.138183    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:01.209311 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.209217    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lm2gp" event={"ID":"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c","Type":"ContainerStarted","Data":"d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3"}
Oct 30 22:01:01.211423 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.211384    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbvwt" event={"ID":"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165","Type":"ContainerStarted","Data":"7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36"}
Oct 30 22:01:01.289740 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.289699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.830868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.830938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.831157    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.831222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.831203688 +0000 UTC m=+39.346776947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.831831    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.831851    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:01.832141 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.831902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.831886299 +0000 UTC m=+39.347459555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.932235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.932339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.932388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.932424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:01.932474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932630    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932710    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.932676126 +0000 UTC m=+39.448249382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932784    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932822    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.932809586 +0000 UTC m=+39.448382842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932893    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.932882911 +0000 UTC m=+39.448456167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : configmap references non-existent config key: service-ca.crt
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932955    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.932984    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.932974349 +0000 UTC m=+39.448547607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.933048    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.933059    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:01.933138 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:01.933088    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.933078931 +0000 UTC m=+39.448652188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:02.033198 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:02.033158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:02.033433 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:02.033379    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:02.033519 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:02.033442    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:06.033423836 +0000 UTC m=+39.548997078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:03.609193 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:03.609127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:03.760326 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:03.760287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:04.246345 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:04.246304    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7" event={"ID":"10e32bef-cfe9-48eb-84e9-44fdebb43090","Type":"ContainerStarted","Data":"882e2cc93f1c9c0232464d5be6ab3ce77fba803613b273a839aff2d98a66a18e"}
Oct 30 22:01:04.507364 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:04.507258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:04.657274 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:04.657177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:05.879028 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.878985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:05.879540 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.879069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:05.879540 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.879344    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:01:05.879540 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.879416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.879396526 +0000 UTC m=+47.394969812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:01:05.879540 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.879498    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:05.879540 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.879512    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:05.879540 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.879540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.879531056 +0000 UTC m=+47.395104310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:05.980583 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.980532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:05.980818 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.980625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:05.980818 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.980706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:05.980818 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.980801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:05.980971 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:05.980849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:05.981019 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.980994    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:05.981078 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981065    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.981043837 +0000 UTC m=+47.496617115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:05.981183 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981148    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:05.981183 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981160    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:05.981278 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981202    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.9811889 +0000 UTC m=+47.496762171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:05.981278 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981264    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:01:05.981374 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.981289772 +0000 UTC m=+47.496863031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:01:05.981374 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981360    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:05.981473 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981398    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.981386004 +0000 UTC m=+47.496959260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:05.981473 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:05.981466    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.981456108 +0000 UTC m=+47.497029364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : configmap references non-existent config key: service-ca.crt
Oct 30 22:01:06.081504 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.081406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:06.081668 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:06.081551    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:06.081668 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:06.081613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.081595706 +0000 UTC m=+47.597168965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:06.256795 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.256759    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cxs8w" event={"ID":"be68f52a-8bf4-474e-b28d-133aecbfb229","Type":"ContainerStarted","Data":"62295b7387f480a1535263f591ed4a87bdfc7e5d42f3bba5abe18a425ea31049"}
Oct 30 22:01:06.258528 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.258505    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/0.log"
Oct 30 22:01:06.258714 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.258541    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" containerID="ffd18e013daded8da793150122100c1ff04344ff18133b68de9f0a64c7ea8180" exitCode=255
Oct 30 22:01:06.258876 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.258818    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" event={"ID":"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b","Type":"ContainerDied","Data":"ffd18e013daded8da793150122100c1ff04344ff18133b68de9f0a64c7ea8180"}
Oct 30 22:01:06.258992 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.258887    2577 scope.go:122] "RemoveContainer" containerID="ffd18e013daded8da793150122100c1ff04344ff18133b68de9f0a64c7ea8180"
Oct 30 22:01:06.261340 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.261314    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh" event={"ID":"f572fd28-f04a-45ed-a153-5921471fb276","Type":"ContainerStarted","Data":"2d8f137ac1aa97f0889730ddccdb7ad0db0759aafa05ce76370dbd569a32b87a"}
Oct 30 22:01:06.263738 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.263712    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s96ng" event={"ID":"914a456e-5637-4208-98bd-ee4078634291","Type":"ContainerStarted","Data":"29d757047ea456eb60eae23fac946660acfbf7f6027e8e2d7b8f3e64dfcfd3e3"}
Oct 30 22:01:06.265758 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.265733    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w" event={"ID":"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4","Type":"ContainerStarted","Data":"e67178adae82286e7154f1d66f4528428b34e82018b335e7e92f0d073c675174"}
Oct 30 22:01:06.378965 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.378913    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-cxs8w" podStartSLOduration=398.074615507 podStartE2EDuration="6m44.378897032s" totalImagesPullingTime="6.304281525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:06.378079939 +0000 UTC m=+39.893653202" watchObservedRunningTime="2026-10-30 22:01:06.378897032 +0000 UTC m=+39.894470309"
Oct 30 22:01:06.379286 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.379255    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7" podStartSLOduration=406.351715725 podStartE2EDuration="6m50.379246327s" totalImagesPullingTime="4.027530602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:04.367584511 +0000 UTC m=+37.883157775" watchObservedRunningTime="2026-10-30 22:01:06.379246327 +0000 UTC m=+39.894819590"
Oct 30 22:01:06.499276 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.498398    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w" podStartSLOduration=402.11598089 podStartE2EDuration="6m48.498382851s" totalImagesPullingTime="6.382401961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:06.497415698 +0000 UTC m=+40.012988961" watchObservedRunningTime="2026-10-30 22:01:06.498382851 +0000 UTC m=+40.013956114"
Oct 30 22:01:06.620859 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.620562    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh" podStartSLOduration=406.353065817 podStartE2EDuration="6m52.62054609s" totalImagesPullingTime="6.267480273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:06.618991239 +0000 UTC m=+40.134564502" watchObservedRunningTime="2026-10-30 22:01:06.62054609 +0000 UTC m=+40.136119353"
Oct 30 22:01:06.759077 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.759037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:06.858973 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.858931    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nxvq5"]
Oct 30 22:01:06.863469 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.863421    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s96ng" podStartSLOduration=3.253050794 podStartE2EDuration="9.863404786s" totalImagesPullingTime="6.610353992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:06.863113197 +0000 UTC m=+40.378686467" watchObservedRunningTime="2026-10-30 22:01:06.863404786 +0000 UTC m=+40.378978050"
Oct 30 22:01:06.920547 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.920462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:06.923200 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.923172    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nxvq5"]
Oct 30 22:01:06.923306 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.923293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:06.981366 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.981323    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:01:06.981639 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.981619    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g5t42\""
Oct 30 22:01:06.981849 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.981619    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:01:06.992092 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.992055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-data-volume\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:06.992229 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.992123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pw5s\" (UniqueName: \"kubernetes.io/projected/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-api-access-7pw5s\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:06.992229 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.992172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-crio-socket\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:06.992229 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.992199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:06.992400 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:06.992228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093432 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-data-volume\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093621 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pw5s\" (UniqueName: \"kubernetes.io/projected/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-api-access-7pw5s\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093621 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-crio-socket\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093621 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093621 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093894 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.093766    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:07.093894 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-data-volume\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093894 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.093836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:07.593816478 +0000 UTC m=+41.109389721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:07.093894 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.093834    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-crio-socket\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.093894 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.093883    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:01:07.094124 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.093917    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:07.593906862 +0000 UTC m=+41.109480103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : configmap "kube-rbac-proxy" not found
Oct 30 22:01:07.100389 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.100367    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-slw2d-b5qs2"]
Oct 30 22:01:07.102665 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.102640    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.116900 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.116876    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.161605 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.161573    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-slw2d-b5qs2"]
Oct 30 22:01:07.167138 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.167100    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pw5s\" (UniqueName: \"kubernetes.io/projected/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-api-access-7pw5s\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.194736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.194639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.194931 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.194752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.194931 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.194802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.194931 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.194842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.258365 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.258340    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"]
Oct 30 22:01:07.274453 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.274430    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/1.log"
Oct 30 22:01:07.275866 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.274926    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/0.log"
Oct 30 22:01:07.275866 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.274966    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" containerID="97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201" exitCode=255
Oct 30 22:01:07.275866 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.275796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.276074 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.275869    2577 scope.go:122] "RemoveContainer" containerID="97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201"
Oct 30 22:01:07.276074 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.276045    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q2lps_openshift-console-operator(e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" podUID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b"
Oct 30 22:01:07.286405 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.286247    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs" event={"ID":"63fef56c-376f-4d60-a2c9-c9b02731c5ed","Type":"ContainerStarted","Data":"1120680e011cc7ad45821e4a257077e72a08ac5a11935208c65e2b44b0c68dd8"}
Oct 30 22:01:07.286405 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.286286    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"]
Oct 30 22:01:07.286405 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.286394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:07.296449 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.296422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcgc\" (UniqueName: \"kubernetes.io/projected/43829ec3-a430-4d2e-b2ee-19ae8df72ac9-kube-api-access-kvcgc\") pod \"migrator-66df5d496b-n8qq6\" (UID: \"43829ec3-a430-4d2e-b2ee-19ae8df72ac9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:07.296561 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.296537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.296841 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.296817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.296931 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.296889    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.297005 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.296938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.297005 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.296998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.297657 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.297633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.297852 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.297832    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.304770 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.304747    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" event={"ID":"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b","Type":"ContainerDied","Data":"97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201"}
Oct 30 22:01:07.304864 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.304792    2577 scope.go:122] "RemoveContainer" containerID="ffd18e013daded8da793150122100c1ff04344ff18133b68de9f0a64c7ea8180"
Oct 30 22:01:07.304922 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.304868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:07.318308 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.318287    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"]
Oct 30 22:01:07.332644 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.332584    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"]
Oct 30 22:01:07.343237 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.343211    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6l5qx\""
Oct 30 22:01:07.344367 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.343413    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:07.344367 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.343423    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:01:07.362507 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.362483    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.397641 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.397604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcgc\" (UniqueName: \"kubernetes.io/projected/43829ec3-a430-4d2e-b2ee-19ae8df72ac9-kube-api-access-kvcgc\") pod \"migrator-66df5d496b-n8qq6\" (UID: \"43829ec3-a430-4d2e-b2ee-19ae8df72ac9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:07.397808 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.397651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptwh\" (UniqueName: \"kubernetes.io/projected/bd3282fe-9d41-4c82-9bac-bb74c4209b62-kube-api-access-8ptwh\") pod \"migrator-66df5d496b-x6s86\" (UID: \"bd3282fe-9d41-4c82-9bac-bb74c4209b62\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:07.400959 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.400899    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs" podStartSLOduration=395.952799027 podStartE2EDuration="6m43.40088169s" totalImagesPullingTime="7.448082663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:07.398941421 +0000 UTC m=+40.914514686" watchObservedRunningTime="2026-10-30 22:01:07.40088169 +0000 UTC m=+40.916454954"
Oct 30 22:01:07.437462 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.437422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.440306 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.440278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:07.461989 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.461959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcgc\" (UniqueName: \"kubernetes.io/projected/43829ec3-a430-4d2e-b2ee-19ae8df72ac9-kube-api-access-kvcgc\") pod \"migrator-66df5d496b-n8qq6\" (UID: \"43829ec3-a430-4d2e-b2ee-19ae8df72ac9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:07.499130 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.499100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptwh\" (UniqueName: \"kubernetes.io/projected/bd3282fe-9d41-4c82-9bac-bb74c4209b62-kube-api-access-8ptwh\") pod \"migrator-66df5d496b-x6s86\" (UID: \"bd3282fe-9d41-4c82-9bac-bb74c4209b62\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:07.564249 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.564222    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptwh\" (UniqueName: \"kubernetes.io/projected/bd3282fe-9d41-4c82-9bac-bb74c4209b62-kube-api-access-8ptwh\") pod \"migrator-66df5d496b-x6s86\" (UID: \"bd3282fe-9d41-4c82-9bac-bb74c4209b62\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:07.599950 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.599908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.600129 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.599961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:07.600129 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.600035    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:01:07.600129 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.600118    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:08.60008407 +0000 UTC m=+42.115657332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : configmap "kube-rbac-proxy" not found
Oct 30 22:01:07.600297 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.600180    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:07.600297 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:07.600246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:08.600228864 +0000 UTC m=+42.115802107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:07.637618 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.637588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:07.645744 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.645720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:07.662756 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.662650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:07.865388 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:01:07.865349    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b72179a_878b_4792_939e_6db6499f00bd.slice/crio-b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf WatchSource:0}: Error finding container b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf: Status 404 returned error can't find the container with id b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf
Oct 30 22:01:07.867707 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.867672    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-slw2d-b5qs2"]
Oct 30 22:01:07.997182 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:07.997151    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jt4tn"]
Oct 30 22:01:08.017351 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.017293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.035708 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.035664    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"]
Oct 30 22:01:08.043723 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:01:08.042811    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43829ec3_a430_4d2e_b2ee_19ae8df72ac9.slice/crio-0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93 WatchSource:0}: Error finding container 0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93: Status 404 returned error can't find the container with id 0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93
Oct 30 22:01:08.060086 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.060035    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jt4tn"]
Oct 30 22:01:08.068382 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:01:08.068346    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3282fe_9d41_4c82_9bac_bb74c4209b62.slice/crio-710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33 WatchSource:0}: Error finding container 710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33: Status 404 returned error can't find the container with id 710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33
Oct 30 22:01:08.071645 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.071627    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"]
Oct 30 22:01:08.076947 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.076924    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 22:01:08.077063 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.077051    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9mpbc\""
Oct 30 22:01:08.077119 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.077109    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 22:01:08.077166 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.077156    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 22:01:08.077229 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.077220    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 22:01:08.104017 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.103985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcks\" (UniqueName: \"kubernetes.io/projected/0199affc-a6e4-46a9-a96f-351e6bbbafa8-kube-api-access-mdcks\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.104194 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.104143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-cabundle\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.104251 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.104236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-key\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.193622 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.193579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:08.205736 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.205704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-cabundle\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.205878 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.205855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-key\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.205954 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.205937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcks\" (UniqueName: \"kubernetes.io/projected/0199affc-a6e4-46a9-a96f-351e6bbbafa8-kube-api-access-mdcks\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.206471 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.206439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-cabundle\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.208898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.208879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-key\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.272472 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.272394    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcks\" (UniqueName: \"kubernetes.io/projected/0199affc-a6e4-46a9-a96f-351e6bbbafa8-kube-api-access-mdcks\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.281203 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.281166    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slw2d-b5qs2" event={"ID":"4b72179a-878b-4792-939e-6db6499f00bd","Type":"ContainerStarted","Data":"edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f"}
Oct 30 22:01:08.281389 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.281228    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slw2d-b5qs2" event={"ID":"4b72179a-878b-4792-939e-6db6499f00bd","Type":"ContainerStarted","Data":"b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf"}
Oct 30 22:01:08.282898 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.282870    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lm2gp" event={"ID":"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c","Type":"ContainerStarted","Data":"b47e0de9f5eca80e5dfd488f70f9cbe5b3ec85d1fa36aa40cebb6122b02be15e"}
Oct 30 22:01:08.283030 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.282928    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:08.284376 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.284345    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86" event={"ID":"bd3282fe-9d41-4c82-9bac-bb74c4209b62","Type":"ContainerStarted","Data":"710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33"}
Oct 30 22:01:08.286016 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.285995    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/1.log"
Oct 30 22:01:08.286334 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.286317    2577 scope.go:122] "RemoveContainer" containerID="97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201"
Oct 30 22:01:08.286558 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:08.286537    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q2lps_openshift-console-operator(e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" podUID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b"
Oct 30 22:01:08.287390 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.287370    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6" event={"ID":"43829ec3-a430-4d2e-b2ee-19ae8df72ac9","Type":"ContainerStarted","Data":"0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93"}
Oct 30 22:01:08.355800 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.355767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:08.367082 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.367034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:08.404288 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.404236    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-slw2d-b5qs2" podStartSLOduration=1.404219433 podStartE2EDuration="1.404219433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.40320678 +0000 UTC m=+41.918780046" watchObservedRunningTime="2026-10-30 22:01:08.404219433 +0000 UTC m=+41.919792697"
Oct 30 22:01:08.495128 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.495097    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:08.495277 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.495170    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:08.609528 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.609455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:08.609528 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.609497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:08.609733 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:08.609579    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:01:08.609733 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:08.609598    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:08.609733 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:08.609643    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:10.609629073 +0000 UTC m=+44.125202333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : configmap "kube-rbac-proxy" not found
Oct 30 22:01:08.609733 ip-10-0-129-125 kubenswrapper[2577]: E1030 22:01:08.609661    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:10.609655031 +0000 UTC m=+44.125228284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:08.667265 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.667210    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lm2gp" podStartSLOduration=35.055047869 podStartE2EDuration="41.667194834s" totalImagesPullingTime="6.612146965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.666603679 +0000 UTC m=+42.182176941" watchObservedRunningTime="2026-10-30 22:01:08.667194834 +0000 UTC m=+42.182768136"
Oct 30 22:01:08.707197 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.707163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:08.759427 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.759404    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jt4tn"]
Oct 30 22:01:08.763393 ip-10-0-129-125 kubenswrapper[2577]: W1030 22:01:08.763363    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0199affc_a6e4_46a9_a96f_351e6bbbafa8.slice/crio-62696e2740a7c874b84c2742cf0714ba067e390e3f6ff8f8736992243326ba71 WatchSource:0}: Error finding container 62696e2740a7c874b84c2742cf0714ba067e390e3f6ff8f8736992243326ba71: Status 404 returned error can't find the container with id 62696e2740a7c874b84c2742cf0714ba067e390e3f6ff8f8736992243326ba71
Oct 30 22:01:08.869577 ip-10-0-129-125 kubenswrapper[2577]: I1030 22:01:08.869549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:09.094267 ip-10-0-129-125 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 22:01:09.108240 ip-10-0-129-125 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 22:01:09.108425 ip-10-0-129-125 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 22:01:09.108479 ip-10-0-129-125 systemd[1]: kubelet.service: Consumed 3.213s CPU time.
Oct 30 22:01:09.118431 ip-10-0-129-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:01:09.194109 ip-10-0-129-125 kubenswrapper[5028]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:01:09.194109 ip-10-0-129-125 kubenswrapper[5028]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:01:09.194109 ip-10-0-129-125 kubenswrapper[5028]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:01:09.194109 ip-10-0-129-125 kubenswrapper[5028]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:01:09.196232 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196139    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.196232 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196232    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196237    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196240    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196244    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196247    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196250    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196253    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196256    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196258    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196261    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196264    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196266    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196269    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196272    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196274    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196277    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196279    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196282    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196284    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196287    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.196304 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196289    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196292    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196294    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196297    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196304    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196306    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196309    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196312    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196315    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196319    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196322    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196325    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196327    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196330    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196333    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196335    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196338    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196340    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196343    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196345    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.196871 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196349    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196351    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196354    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196356    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196359    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196361    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196364    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196366    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196369    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196371    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196374    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196377    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196379    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196382    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196385    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196388    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196390    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196394    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196396    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.197421 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196399    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196402    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196404    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196408    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196410    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196413    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196416    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196418    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196421    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196423    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196425    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196428    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196431    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196433    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196436    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196438    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196441    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196443    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196446    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196448    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.197885 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196450    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196453    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196455    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196458    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196461    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196463    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196466    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196469    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196471    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196474    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196476    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196480    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196482    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196485    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196488    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196491    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196493    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196496    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196499    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.198568 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196504    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196507    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196511    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196514    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196516    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196521    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.196525    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197329    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197337    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197340    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197344    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197346    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197349    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197353    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197358    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197361    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197364    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197367    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197370    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.199032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197372    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197375    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197377    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197381    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197383    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197386    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197389    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197395    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197399    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197401    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197404    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197407    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197409    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197412    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197414    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197417    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197419    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197422    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197425    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.199475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197427    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197430    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197433    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197436    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197438    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197441    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197443    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197446    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197449    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197451    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197454    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197456    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197459    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197462    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197464    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197467    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197470    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197472    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197475    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197477    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.199967 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197480    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197482    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197485    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197487    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197491    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197493    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197495    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197498    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197501    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197503    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197505    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197508    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197510    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197512    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197515    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197517    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197519    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197522    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197524    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197527    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.200481 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197530    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197532    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197535    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197537    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197540    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197542    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197545    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197548    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197551    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197553    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197556    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197558    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197560    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197563    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197566    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197568    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197571    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197573    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197576    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197578    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.201032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197581    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197583    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197585    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197588    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197591    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197593    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197596    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197598    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197601    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197603    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197606    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197608    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197611    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197613    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.197615    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198130    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198136    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198140    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198144    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.201538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198147    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198150    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198152    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198155    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198158    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198160    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198164    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198167    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198169    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198172    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198175    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198178    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198180    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198183    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198185    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198188    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198190    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198193    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198196    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198199    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198202    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.202032 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198204    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198207    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198209    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198212    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198214    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198217    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198220    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198222    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198226    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198230    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198232    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198235    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198238    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198241    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198243    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198246    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198248    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198251    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198254    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.202736 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198256    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198259    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198261    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198264    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198266    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198269    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198272    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198274    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198277    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198279    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198281    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198284    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198286    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198289    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198291    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198294    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198296    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198299    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198301    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.203516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198303    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198306    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198308    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198311    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198316    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198322    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198325    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198329    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198332    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198334    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198337    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198339    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198342    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198345    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198347    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198350    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198353    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198355    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198358    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.203987 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198360    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198363    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198366    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198368    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198371    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198373    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198376    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198378    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198380    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198383    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198385    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198388    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198391    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198393    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198396    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198399    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198402    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198406    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198408    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198411    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.204475 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198413    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198416    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198420    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.198423    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198510    5028 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198521    5028 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198526    5028 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198532    5028 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198537    5028 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198541    5028 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198545    5028 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198549    5028 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198553    5028 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198556    5028 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198560    5028 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198563    5028 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198566    5028 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198569    5028 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198572    5028 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198575    5028 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198578    5028 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198581    5028 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198585    5028 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:01:09.204976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198588    5028 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198592    5028 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198595    5028 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198598    5028 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198602    5028 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198605    5028 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198609    5028 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198612    5028 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198615    5028 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198618    5028 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198621    5028 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198625    5028 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198628    5028 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198632    5028 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198635    5028 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198638    5028 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198641    5028 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198645    5028 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198648    5028 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198652    5028 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198655    5028 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198658    5028 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198661    5028 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198664    5028 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:01:09.205538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198668    5028 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198671    5028 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198674    5028 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198678    5028 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198681    5028 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198700    5028 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198705    5028 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198709    5028 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198714    5028 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198718    5028 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198723    5028 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198729    5028 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198734    5028 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198738    5028 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198741    5028 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198743    5028 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198746    5028 flags.go:64] FLAG: --help="false"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198749    5028 flags.go:64] FLAG: --hostname-override="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198753    5028 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198755    5028 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198758    5028 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198761    5028 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198765    5028 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:01:09.206116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198768    5028 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198771    5028 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198774    5028 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198777    5028 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198780    5028 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198784    5028 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198787    5028 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198789    5028 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198792    5028 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198796    5028 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198799    5028 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198801    5028 flags.go:64] FLAG: --lock-file=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198804    5028 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198807    5028 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198810    5028 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198815    5028 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198818    5028 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198821    5028 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198824    5028 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198827    5028 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198830    5028 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198833    5028 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198836    5028 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198841    5028 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198844    5028 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:01:09.206643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198847    5028 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198850    5028 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198853    5028 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198856    5028 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198859    5028 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198862    5028 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198864    5028 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198867    5028 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198874    5028 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198877    5028 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198880    5028 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198884    5028 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198887    5028 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198890    5028 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198893    5028 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198896    5028 flags.go:64] FLAG: --port="10250"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198899    5028 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198902    5028 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0114d9045ac87c8f8"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198905    5028 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198908    5028 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198910    5028 flags.go:64] FLAG: --register-node="true"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198913    5028 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198917    5028 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198920    5028 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198922    5028 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198925    5028 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:01:09.207252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198929    5028 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198932    5028 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198935    5028 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198938    5028 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198941    5028 flags.go:64] FLAG: --runonce="false"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198944    5028 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198949    5028 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198953    5028 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198956    5028 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198958    5028 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198961    5028 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198964    5028 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198968    5028 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198971    5028 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198974    5028 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198977    5028 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198980    5028 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198982    5028 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198987    5028 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198990    5028 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198996    5028 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.198998    5028 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199001    5028 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199005    5028 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199008    5028 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:01:09.207922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199011    5028 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199014    5028 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199016    5028 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199020    5028 flags.go:64] FLAG: --v="2"
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199024    5028 flags.go:64] FLAG: --version="false"
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199028    5028 flags.go:64] FLAG: --vmodule=""
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199031    5028 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199035    5028 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199161    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199164    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199167    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199170    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199172    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199177    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199180    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199182    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199185    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199187    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199190    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199192    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199195    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.208540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199198    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199200    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199203    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199205    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199209    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199215    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199218    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199221    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199223    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199226    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199228    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199230    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199233    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199236    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199238    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199240    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199243    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199245    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199248    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199251    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.209137 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199253    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199255    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199258    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199261    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199266    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199269    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199271    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199274    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199277    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199280    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199282    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199285    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199287    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199291    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199295    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199298    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199302    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199304    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199307    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.209652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199310    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199313    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199315    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199318    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199321    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199325    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199328    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199331    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199334    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199337    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199340    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199342    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199345    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199347    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199350    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199352    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199355    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199359    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199362    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.210357 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199364    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199367    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199369    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199372    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199374    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199377    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199379    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199382    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199384    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199387    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199391    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199393    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199396    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199399    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199401    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199404    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199406    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199409    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199411    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199414    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.211083 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199416    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199419    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199421    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199424    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199426    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199429    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199431    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199434    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199437    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199439    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199443    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199446    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199448    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199451    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199453    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.199477    5028 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:01:09.211821 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199512    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199517    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199521    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199524    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199526    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199529    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199533    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199536    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199538    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199541    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199544    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199547    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199549    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199552    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199555    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199557    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199559    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199562    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199565    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199567    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.212308 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199570    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199572    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199575    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199577    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199580    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199582    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199585    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199589    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199592    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199594    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199597    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199600    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199602    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199605    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199607    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199610    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199612    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199615    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199619    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199621    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.213250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199624    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199626    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199629    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199632    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199635    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199638    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199640    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199642    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199645    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199647    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199650    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199652    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199655    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199657    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199660    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199662    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199665    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199667    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199670    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.214119 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199673    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199676    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199678    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199681    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199696    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199698    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199701    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199704    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199707    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199709    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199712    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199716    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199720    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199724    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199727    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199730    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199733    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199736    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199739    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.214878 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199741    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199744    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199746    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199749    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199753    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199757    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199760    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199763    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199766    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199769    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199772    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199775    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199778    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199782    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199785    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199788    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199791    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199793    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199796    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.215701 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199798    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199801    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199803    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199806    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199809    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199813    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199816    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199819    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.199822    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.204825    5028 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.204842    5028 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.204862    5028 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.204868    5028 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204927    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204932    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204935    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204938    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204941    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204944    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204948    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.216279 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204951    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204954    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204958    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204961    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204964    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204967    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204970    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204973    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204976    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204979    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204983    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204986    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204988    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204991    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204994    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204996    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.204999    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205002    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205004    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205007    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.216802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205010    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205012    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205015    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205018    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205021    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205025    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205029    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205034    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205037    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205039    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205042    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205045    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205048    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205051    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205054    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205056    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205059    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205062    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205064    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.217289 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205068    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205071    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205074    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205076    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205079    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205082    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205084    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205087    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205090    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205093    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205095    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205098    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205101    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205104    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205107    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205110    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205112    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205115    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205118    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205120    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.217777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205123    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205126    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205128    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205131    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205134    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205136    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205139    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205142    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205144    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205147    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205149    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205152    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205154    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205158    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205161    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205164    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205166    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205169    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205171    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.218234 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205174    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205177    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205180    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205182    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205185    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205187    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205190    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205193    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205195    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205198    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205200    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205204    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205208    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205211    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205214    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205216    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205219    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205222    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205224    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205227    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.218668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205229    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.205257    5028 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205417    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205422    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205425    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205428    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205430    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205434    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205439    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205442    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205445    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205448    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205450    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205453    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205455    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.219157 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205458    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205461    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205463    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205466    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205468    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205471    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205473    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205476    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205478    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205481    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205484    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205486    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205488    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205491    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205493    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205496    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205498    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205501    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205503    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.219511 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205506    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205508    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205511    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205513    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205516    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205518    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205521    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205524    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205527    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205529    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205532    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205535    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205537    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205540    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205543    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205545    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205548    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205551    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205553    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205556    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.220142 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205559    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205561    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205564    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205566    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205569    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205571    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205574    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205576    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205578    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205581    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205583    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205586    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205588    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205591    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205594    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205596    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205600    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205603    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205606    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.220868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205609    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205612    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205615    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205617    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205620    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205622    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205624    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205627    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205629    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205632    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205634    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205636    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205639    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205641    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205643    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205646    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205648    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205651    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205653    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205656    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.221651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205658    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205661    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205664    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205666    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205668    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205671    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205673    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205676    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205678    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205681    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205696    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205699    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205701    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205705    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205708    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.205731    5028 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:01:09.222309 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205762    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205766    5028 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205769    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205772    5028 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205774    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205777    5028 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205779    5028 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205782    5028 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205785    5028 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205787    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205789    5028 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205792    5028 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205794    5028 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205797    5028 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205800    5028 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205802    5028 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205804    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205807    5028 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205809    5028 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205812    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:01:09.222750 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205814    5028 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205817    5028 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205820    5028 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205822    5028 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205825    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205828    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205830    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205833    5028 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205835    5028 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205838    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205841    5028 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205844    5028 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205846    5028 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205849    5028 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205852    5028 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205854    5028 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205856    5028 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205859    5028 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205862    5028 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205864    5028 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:01:09.223243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205867    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205869    5028 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205871    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205874    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205876    5028 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205879    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205881    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205884    5028 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205886    5028 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205889    5028 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205891    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205895    5028 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205899    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205902    5028 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205904    5028 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205907    5028 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205909    5028 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205912    5028 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205914    5028 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:01:09.223777 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205917    5028 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205920    5028 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205922    5028 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205928    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205931    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205933    5028 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205936    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205938    5028 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205940    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205943    5028 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205945    5028 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205948    5028 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205950    5028 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205952    5028 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205955    5028 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205959    5028 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205962    5028 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205964    5028 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205967    5028 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205970    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:01:09.224243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205972    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205975    5028 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205978    5028 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205980    5028 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205983    5028 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205986    5028 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205988    5028 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205991    5028 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205993    5028 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205996    5028 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.205998    5028 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206001    5028 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206003    5028 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206006    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206008    5028 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206013    5028 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206016    5028 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206019    5028 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206022    5028 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:01:09.224729 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206025    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206028    5028 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206030    5028 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206033    5028 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206036    5028 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206039    5028 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206041    5028 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:09.206044    5028 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.206103    5028 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207058    5028 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207118    5028 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207306    5028 server.go:1014] "Starting client certificate rotation"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207448    5028 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207501    5028 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:55:27 +0000 UTC" deadline="2028-07-26 02:26:12.61880848 +0000 UTC"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207526    5028 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15220h25m3.411284365s"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.207805    5028 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:01:09.225202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.208935    5028 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:01:09.225614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.212158    5028 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:01:09.225614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.214458    5028 log.go:25] "Validated CRI v1 image API"
Oct 30 22:01:09.225614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.215339    5028 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:01:09.225614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.217443    5028 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8de244e1-938a-4c1d-9bff-333a7a861980:/dev/nvme0n1p3 e3fa749d-938e-473d-8bd1-cc2c25ceb39f:/dev/nvme0n1p4]
Oct 30 22:01:09.225761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.217468    5028 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15ca7cf28021ca7e84c8963fb04888f391883b955c4a76a5dcd0b12a48a5186f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15ca7cf28021ca7e84c8963fb04888f391883b955c4a76a5dcd0b12a48a5186f/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62696e2740a7c874b84c2742cf0714ba067e390e3f6ff8f8736992243326ba71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62696e2740a7c874b84c2742cf0714ba067e390e3f6ff8f8736992243326ba71/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0199affc-a6e4-46a9-a96f-351e6bbbafa8/volumes/kubernetes.io~projected/kube-api-access-mdcks:{mountpoint:/var/lib/kubelet/pods/0199affc-a6e4-46a9-a96f-351e6bbbafa8/volumes/kubernetes.io~projected/kube-api-access-mdcks major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0199affc-a6e4-46a9-a96f-351e6bbbafa8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0199affc-a6e4-46a9-a96f-351e6bbbafa8/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03d17bd7-a7c6-45b9-b0e5-2d9809543781/volumes/kubernetes.io~projected/kube-api-access-n2shr:{mountpoint:/var/lib/kubelet/pods/03d17bd7-a7c6-45b9-b0e5-2d9809543781/volumes/kubernetes.io~projected/kube-api-access-n2shr major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~projected/kube-api-access-7nkvm:{mountpoint:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~projected/kube-api-access-7nkvm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10e32bef-cfe9-48eb-84e9-44fdebb43090/volumes/kubernetes.io~projected/kube-api-access-q4sd9:{mountpoint:/var/lib/kubelet/pods/10e32bef-cfe9-48eb-84e9-44fdebb43090/volumes/kubernetes.io~projected/kube-api-access-q4sd9 major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a535db2-2694-4965-b4f9-5b478681381e/volumes/kubernetes.io~projected/kube-api-access-lxt6q:{mountpoint:/var/lib/kubelet/pods/1a535db2-2694-4965-b4f9-5b478681381e/volumes/kubernetes.io~projected/kube-api-access-lxt6q major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dc81215-458c-47bb-87e8-ed1893729907/volumes/kubernetes.io~projected/kube-api-access-4545h:{mountpoint:/var/lib/kubelet/pods/1dc81215-458c-47bb-87e8-ed1893729907/volumes/kubernetes.io~projected/kube-api-access-4545h major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9/volumes/kubernetes.io~projected/kube-api-access-7pw5s:{mountpoint:/var/lib/kubelet/pods/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9/volumes/kubernetes.io~projected/kube-api-access-7pw5s major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43829ec3-a430-4d2e-b2ee-19ae8df72ac9/volumes/kubernetes.io~projected/kube-api-access-kvcgc:{mountpoint:/var/lib/kubelet/pods/43829ec3-a430-4d2e-b2ee-19ae8df72ac9/volumes/kubernetes.io~projected/kube-api-access-kvcgc major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b72179a-878b-4792-939e-6db6499f00bd/volumes/kubernetes.io~projected/kube-api-access-p2fzt:{mountpoint:/var/lib/kubelet/pods/4b72179a-878b-4792-939e-6db6499f00bd/volumes/kubernetes.io~projected/kube-api-access-p2fzt major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cafb6fd-d8bd-417e-8b93-d42dbd70d102/volumes/kubernetes.io~projected/kube-api-access-qhcf9:{mountpoint:/var/lib/kubelet/pods/4cafb6fd-d8bd-417e-8b93-d42dbd70d102/volumes/kubernetes.io~projected/kube-api-access-qhcf9 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e8b0fdc-6162-4680-b22b-d1eb1656f35b/volumes/kubernetes.io~projected/kube-api-access-qxhgf:{mountpoint:/var/lib/kubelet/pods/4e8b0fdc-6162-4680-b22b-d1eb1656f35b/volumes/kubernetes.io~projected/kube-api-access-qxhgf major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5240e9dc-63a3-436f-af94-ab88f6af0c36/volumes/kubernetes.io~projected/kube-api-access-5jt6r:{mountpoint:/var/lib/kubelet/pods/5240e9dc-63a3-436f-af94-ab88f6af0c36/volumes/kubernetes.io~projected/kube-api-access-5jt6r major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~projected/kube-api-access-dx9wd:{mountpoint:/var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~projected/kube-api-access-dx9wd major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~secret/stats-auth major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63fef56c-376f-4d60-a2c9-c9b02731c5ed/volumes/kubernetes.io~projected/kube-api-access-6kqtv:{mountpoint:/var/lib/kubelet/pods/63fef56c-376f-4d60-a2c9-c9b02731c5ed/volumes/kubernetes.io~projected/kube-api-access-6kqtv major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volumes/kubernetes.io~projected/kube-api-access-2kxzk:{mountpoint:/var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volumes/kubernetes.io~projected/kube-api-access-2kxzk major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e/volumes/kubernetes.io~projected/kube-api-access-zg4n7:{mountpoint:/var/lib/kubelet/pods/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e/volumes/kubernetes.io~projected/kube-api-access-zg4n7 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/767b8f6d-306f-45ef-97cb-a5e04e233892/volumes/kubernetes.io~projected/kube-api-access-86rms:{mountpoint:/var/lib/kubelet/pods/767b8f6d-306f-45ef-97cb-a5e04e233892/volumes/kubernetes.io~projected/kube-api-access-86rms major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~projected/kube-api-access-cnnjx:{mountpoint:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~projected/kube-api-access-cnnjx major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/914a456e-5637-4208-98bd-ee4078634291/volumes/kubernetes.io~projected/kube-api-access-q28g6:{mountpoint:/var/lib/kubelet/pods/914a456e-5637-4208-98bd-ee4078634291/volumes/kubernetes.io~projected/kube-api-access-q28g6 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/914a456e-5637-4208-98bd-ee4078634291/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/914a456e-5637-4208-98bd-ee4078634291/volumes/kubernetes.io~secret/cert major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c/volumes/kubernetes.io~projected/kube-api-access-cfccn:{mountpoint:/var/lib/kubelet/pods/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c/volumes/kubernetes.io~projected/kube-api-access-cfccn major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a461f5f1-641c-4f5e-8f29-995036f2f0f7/volumes/kubernetes.io~projected/kube-api-access-ld84k:{mountpoint:/var/lib/kubelet/pods/a461f5f1-641c-4f5e-8f29-995036f2f0f7/volumes/kubernetes.io~projected/kube-api-access-ld84k major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4/volumes/kubernetes.io~projected/kube-api-access-p47rc:{mountpoint:/var/lib/kubelet/pods/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4/volumes/kubernetes.io~projected/kube-api-access-p47rc major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3282fe-9d41-4c82-9bac-bb74c4209b62/volumes/kubernetes.io~projected/kube-api-access-8ptwh:{mountpoint:/var/lib/kubelet/pods/bd3282fe-9d41-4c82-9bac-bb74c4209b62/volumes/kubernetes.io~projected/kube-api-access-8ptwh major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be68f52a-8bf4-474e-b28d-133aecbfb229/volumes/kubernetes.io~projected/kube-api-access-slx6l:{mountpoint:/var/lib/kubelet/pods/be68f52a-8bf4-474e-b28d-133aecbfb229/volumes/kubernetes.io~projected/kube-api-access-slx6l major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be68f52a-8bf4-474e-b28d-133aecbfb229/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/be68f52a-8bf4-474e-b28d-133aecbfb229/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd533e21-2bdf-4973-99f5-568d4a5b0922/volumes/kubernetes.io~projected/kube-api-access-x7z47:{mountpoint:/var/lib/kubelet/pods/cd533e21-2bdf-4973-99f5-568d4a5b0922/volumes/kubernetes.io~projected/kube-api-access-x7z47 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28a3c9a-7726-406c-bfdf-f36065f02a81/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d28a3c9a-7726-406c-bfdf-f36065f02a81/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~projected/kube-api-access-4rzpr:{mountpoint:/var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~projected/kube-api-access-4rzpr major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d91ca99d-6276-4875-9d6b-e1e49533a930/volumes/kubernetes.io~projected/kube-api-access-92d4g:{mountpoint:/var/lib/kubelet/pods/d91ca99d-6276-4875-9d6b-e1e49533a930/volumes/kubernetes.io~projected/kube-api-access-92d4g major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/volumes/kubernetes.io~projected/kube-api-access-glgwv:{mountpoint:/var/lib/kubelet/pods/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/volumes/kubernetes.io~projected/kube-api-access-glgwv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/volumes/kubernetes.io~secret/serving-cert major:0 minor:187 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7f71a60-c64f-4960-934a-6c75e8d94546/volumes/kubernetes.io~projected/kube-api-access-vwgfj:{mountpoint:/var/lib/kubelet/pods/e7f71a60-c64f-4960-934a-6c75e8d94546/volumes/kubernetes.io~projected/kube-api-access-vwgfj major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf/volumes/kubernetes.io~projected/kube-api-access-mbrzb:{mountpoint:/var/lib/kubelet/pods/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf/volumes/kubernetes.io~projected/kube-api-access-mbrzb major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f572fd28-f04a-45ed-a153-5921471fb276/volumes/kubernetes.io~projected/kube-api-access-2qvqc:{mountpoint:/var/lib/kubelet/pods/f572fd28-f04a-45ed-a153-5921471fb276/volumes/kubernetes.io~projected/kube-api-access-2qvqc major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f572fd28-f04a-45ed-a153-5921471fb276/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f572fd28-f04a-45ed-a153-5921471fb276/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7eb38a6a06da02e04f37cc44870c775ef5e08bf7e6b64ec608d874c96a8b13ca/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a8666771637a2df268bffcb082350b1ff04fba7a5ed25cfd75ef39fef79ff297/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ac25dc567dae6bacee8ff01faa7b6b922cdf23e70117669303783fa3c5e45052/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/e151136cf8136d2d8d6ef6b89f007406a6dd712dc306220f983bf5253ddd03b0/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a28469f53dd18ddd1c938295b566974b6f47e6759059caa1b3b3ee0b0c21c736/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4fa4956119a338e6674e03735170450d2fe30e7731820b55619e23d7f2872077/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/7101e6dcd2dae6ed5443fbad14ff4ce5ed7548c18bcfbf51f854b0c77bbcb625/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/99efe0f23366f9a2894e813c0d29e6bea4fdfa0998efabaf5546273efae56c3b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e5d5e7e16b3cd9e2e2ff620ac9f7cd03c83dfc02ab1f66999378c0ff35516501/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5f1dc238c72febfa9daf24cb35c6025510af819b42bca17873c3a8ed85832cf3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/22d4e31fb7e90732dd6f6da9e2f2223610c65bd16d8677283e746a23af5649b6/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/89765caff95f85c7af3028411804c978f0f1e523f5caac0cf4b39b7052faf358/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/822350bce076fec2e427da11bd07dabab94fa484c380dff148e47c39fed86cb0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c7522b8265529bb3818ce8839ea7728db6a33385be3a381618b8e50b3ec55625/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/097f9d8916b319075dc5e53db12fb456b4bb12f17713169626f3e112f0dba5f6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c66fa9e10bb096fffee26d9c90c6c6474e37dc02c6ad289a49919cf5eb6f6d66/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5d4be0a2687f193693c29647888b3f594f9e9f0110941aa65c5437b67bbaced6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7f5e53fa8b97f083053ab4147ecd24c997ed50dfb3487355b2a9a88eb1b8275d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0a577aa0676f9d573fc9c6ce847bb2d02553d26d71e088ebb1302d26f5d26665/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/6f23935b59113a9cab69426c49d63ca548e0305a175461ff76af18fa10c6a13c/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/0b83cedc49102ada78554a812c9e142dd5da87967ea371edf55d5364c8db6082/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/1a712b2949f9b5716b100a23fde265f11a69f09076885282ef23080b69201ff0/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/220b6c5347b52742ba5da2c953c95400d90424635d4a687ba2c2e0e425f7f589/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/52c78f15ee623c51da29e17c60a8b3badea995f6fc30f5dc33d82c0fb8ec50cd/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0832ce9434fede52b7993525bb2697e388af34ebaf14b2862230f9ff0e67e6a0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/be6b53434c42b9d810a70933b104fafd2ec7b84a083be9f02cac77a20a3f9c41/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/9a71b590f3ad118396b576e8cce6e12b420eab73297405fa8e13a67d809675d9/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/7548f82ab779326dea6f81a061d9028bf442bd91755603edd190ca8f8f388784/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/a513c501bd50066772f9314c5f9b0bf8f70b0676b10238d87d158a1bde479709/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/95f62e89e3de80a2e633b844415b3130ea87b6ddc303940ce651ba75b565b5e3/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/12c8818895e59169bc2aedb4c36447b220867d55bdc8a0f3f5422d1b254bd597/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/f2dd7df50517082011333b69913501ecead968b801971234eba3ab2673c636b7/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/d4ec233d964fd27a796e709a158f3b7da8b5f6e5bf0c6dff5c60390fdfdd269f/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/4a6a24e1030ef5c55e15478e96dd155ba061123d4148524d7da4fe4748a666e8/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/a443d6b453dd153ee669f465f0783ac72656e292f2fb4154813c1867c0787fed/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/a931aeceb8ce38f4a51b9a95b1d5e6eaf1eea897eb70c1189018ff087eb5b84f/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/c465c772e166d381a26425c8940f49239a32079c3186734cf6fb6ea2aa9abec6/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/cd32f58a0cab109550a49b571e9c7d85813d6d38587099d8741ef4758a7c9760/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/dd27bc385ce2e7aa6aba5402cd3a915d9bd390c5e7c3b4902cc867f0dde22114/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/afc559b0ddd345965947b2a04c6f684c29091e74a00d872e3cef27942773f03b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/cbc6bd47df1fb484a01323c7a986ec94d79dda596ab4a6e13911a0f26796309d/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/10a5f4bf6cc15e79712a38252712633d5cbe9aae832cab79ab9e75f5cb7b48f5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/be08f1affacc16b9320b761fadbc16f04677124c74dd30bfbb5dd2d7f9898902/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/3fa81fd4c83574e767acfdb9dd4f3278d234ddaa218364b8979011784da6d1c8/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/3b1e96bacb2774cf588ad430272cd4c3c625ce5b12bf9c63fc0cd8e52477bc5c/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/06d56d9cca2277b18bdabdb2f99713dc1db30ab4b357ac9cf77c7b3f16148ee1/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/774e2370d5e580a9da00ec514602b7ee720782cf7c73e970357ba793918a239a/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/dec33f99134334d6e41de7a8131f7a0b68d93c6f752809fcb9ca1c02735be82f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/7635d03c96653939df186208d81bf2e810b33b59e25a937beab02bb757ea371a/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/ca9f8dabbfe805b434b21e82edd8d408f5e2b7c40abae788502f5795d34bf0f0/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ca7f50eff0fe1d2145a2c39433d62952cdf365685f4654d2405c88fb19662659/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/2f60910b8212ab35f5a3605c16e8725519faac36c065b50aefbed0c1ae64f127/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8a2a8048ac31f80f7328a3e15e28e35822918be87c5c651a358c69ed2af28cd0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f19759a2c25390bee9e05aa4da1c211558b5916d25e872c73217e75c3b2e4d73/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/361e01b685fc4c0fa8f77ba74fac8b2193a6754320011a3ab848ce06da7e8552/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/062f780d764d92d074e2a88b8e33e165947d2f6a201692b0ed123a36e163bc47/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f755d1104907d27258b746e137d6c3c91bfa1a85cb62ede4302157dbbe79e605/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ff6a8dc08a706befd6ca192a1722be1633867c62015cacee4b30aa085c7e558f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/d72cf4ad5cb4e847c4e1bf2215feb51aa8c12823dfacb0c6eeda63d40f42b081/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/c2e04837d77e7e65b20a094082c0f0c4b2143ffad1bf78872085cde9009a31bb/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 22:01:09.227123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.226754    5028 manager.go:217] Machine: {Timestamp:2026-10-30 22:01:09.226293589 +0000 UTC m=+0.075806551 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3138691 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec205f69f1c74a1a12eeb5cc45ce1a84 SystemUUID:ec205f69-f1c7-4a1a-12ee-b5cc45ce1a84 BootID:b3af563f-81f9-42f8-8625-1835a9ff901e Filesystems:[{Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~projected/kube-api-access-7nkvm DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f572fd28-f04a-45ed-a153-5921471fb276/volumes/kubernetes.io~projected/kube-api-access-2qvqc DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97d616c887e7ddef6bd5011467b9ab4530b1912d2bbf1da92f3c8acee22fc4e9/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/767b8f6d-306f-45ef-97cb-a5e04e233892/volumes/kubernetes.io~projected/kube-api-access-86rms DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03d17bd7-a7c6-45b9-b0e5-2d9809543781/volumes/kubernetes.io~projected/kube-api-access-n2shr DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a26dcda897e9523153eea48a7cebfeac97c383fc67379c5051b4fb46b7f5b085/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5240e9dc-63a3-436f-af94-ab88f6af0c36/volumes/kubernetes.io~projected/kube-api-access-5jt6r DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43829ec3-a430-4d2e-b2ee-19ae8df72ac9/volumes/kubernetes.io~projected/kube-api-access-kvcgc DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62696e2740a7c874b84c2742cf0714ba067e390e3f6ff8f8736992243326ba71/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15ca7cf28021ca7e84c8963fb04888f391883b955c4a76a5dcd0b12a48a5186f/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52ffc008ae35d6a502cdf73e982c08601e4f9e291312cc29c752a56f4a392d5c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3282fe-9d41-4c82-9bac-bb74c4209b62/volumes/kubernetes.io~projected/kube-api-access-8ptwh DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8ce080736d993092ec1d429186c0b07aef606a576feb1cbc8a7ac383872ae1e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/914a456e-5637-4208-98bd-ee4078634291/volumes/kubernetes.io~projected/kube-api-access-q28g6 DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63fef56c-376f-4d60-a2c9-c9b02731c5ed/volumes/kubernetes.io~projected/kube-api-access-6kqtv DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35ecf5cbb030f3eb9fe109f44472c56c9cf24ed030b2e13c81ea7894478da217/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e8b0fdc-6162-4680-b22b-d1eb1656f35b/volumes/kubernetes.io~projected/kube-api-access-qxhgf DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd533e21-2bdf-4973-99f5-568d4a5b0922/volumes/kubernetes.io~projected/kube-api-access-x7z47 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc01af247e90d4e6606af8d9519a0ff3b7d8ac2c74177a562b97b8dbc41e9514/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:187 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a461f5f1-641c-4f5e-8f29-995036f2f0f7/volumes/kubernetes.io~projected/kube-api-access-ld84k DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~projected/kube-api-access-cnnjx DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/1dc81215-458c-47bb-87e8-ed1893729907/volumes/kubernetes.io~projected/kube-api-access-4545h DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0e261fac14527cb44722edc46aa2e53d5deb1733f08c0288d544d476a2b65ba/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4/volumes/kubernetes.io~projected/kube-api-access-p47rc DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e/volumes/kubernetes.io~projected/kube-api-access-zg4n7 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c/volumes/kubernetes.io~projected/kube-api-access-cfccn DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0199affc-a6e4-46a9-a96f-351e6bbbafa8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cafb6fd-d8bd-417e-8b93-d42dbd70d102/volumes/kubernetes.io~projected/kube-api-access-qhcf9 DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83c33392d479cf8b3b1d4440cce71146063c7816d9b1dbd8cb049edbdbf7ac27/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c5c188fedd1f8ae51d653e35c95d77095417e0a5251b4bccae0c8abe878d7a2/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/volumes/kubernetes.io~projected/kube-api-access-glgwv DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d91ca99d-6276-4875-9d6b-e1e49533a930/volumes/kubernetes.io~projected/kube-api-access-92d4g DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf/volumes/kubernetes.io~projected/kube-api-access-mbrzb DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0d1b759a87606bc9cb76c32d412dee49a6b7918a1e1299c6b8935a4baddc4f3/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/914a456e-5637-4208-98bd-ee4078634291/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9/volumes/kubernetes.io~projected/kube-api-access-7pw5s DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f572fd28-f04a-45ed-a153-5921471fb276/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35a20aefdc48ccf0d0d11715c43e7c373639ec0f6abbde3c7af130b0f0a04850/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0199affc-a6e4-46a9-a96f-351e6bbbafa8/volumes/kubernetes.io~projected/kube-api-access-mdcks DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d28a3c9a-7726-406c-bfdf-f36065f02a81/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd24f9cdcbd495cabbac0f480f2ffe795372a7a38685ab5b9c2dc09b5ff4b8d5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10e32bef-cfe9-48eb-84e9-44fdebb43090/volumes/kubernetes.io~projected/kube-api-access-q4sd9 DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7195635265f2d7bb1cb706b1a664feef39902dfdb214b12d6ebac9f652e454af/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d8dd5009160ef81a6e25d760b2879ee40484c4ff79f53e2cb7e26c95eb70318/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be68f52a-8bf4-474e-b28d-133aecbfb229/volumes/kubernetes.io~projected/kube-api-access-slx6l DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/710936dd676061698c48f081092819892de07a78419631e3a1d5b4f15ba2fe33/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/699cbaa140c84869cdb28e2136fbf5f55187ec44bec4f6415b82ecf8d103a0d3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ec5482ed8c78a2b3969a5c423a267b23a7ed22807150dac0d2b1ad4316f46e8/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64039614-3fbe-4a3a-b80f-2078ae1cc041/volumes/kubernetes.io~projected/kube-api-access-2kxzk DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~projected/kube-api-access-4rzpr DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a535db2-2694-4965-b4f9-5b478681381e/volumes/kubernetes.io~projected/kube-api-access-lxt6q DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~projected/kube-api-access-dx9wd DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7d51ef63b97ea22130777c2519e0df63214ee867df790fe46977683d802b11a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e59575bbb5e06d0fb2129e47f8dc460b36e8bf895483652abfa3715f8d72a36/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be68f52a-8bf4-474e-b28d-133aecbfb229/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b72179a-878b-4792-939e-6db6499f00bd/volumes/kubernetes.io~projected/kube-api-access-p2fzt DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7f71a60-c64f-4960-934a-6c75e8d94546/volumes/kubernetes.io~projected/kube-api-access-vwgfj DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56a790e5-683e-4dfd-82e1-1bde4629e09f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d824b6ee-c212-4bea-8c42-5dbd4e57ad60/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab77f30a26e6dbee073dfd55335b46eb49077c1b81e360096b033757b514725a/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0867eb4872c0b7a88fe193d1a40aed173cfcb83caf8ce4b730f365db1e846a93/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0867eb4872c0b7a MacAddress:fe:48:2f:80:44:16 Speed:10000 Mtu:8901} {Name:35ecf5cbb030f3e MacAddress:2e:a5:d0:6d:58:3e Speed:10000 Mtu:8901} {Name:3d8dd5009160ef8 MacAddress:32:71:a7:10:04:9d Speed:10000 Mtu:8901} {Name:3ec5482ed8c78a2 MacAddress:be:83:fa:b3:b6:a4 Speed:10000 Mtu:8901} {Name:52ffc008ae35d6a MacAddress:4e:db:67:71:4d:6d Speed:10000 Mtu:8901} {Name:62696e2740a7c87 MacAddress:ee:de:e3:48:8a:a2 Speed:10000 Mtu:8901} {Name:710936dd6760616 MacAddress:0e:e0:98:33:4c:88 Speed:10000 Mtu:8901} {Name:7195635265f2d7b MacAddress:2a:74:cb:43:99:e1 Speed:10000 Mtu:8901} {Name:7e59575bbb5e06d MacAddress:16:61:4b:54:6b:2b Speed:10000 Mtu:8901} {Name:a26dcda897e9523 MacAddress:4a:19:1b:91:31:20 Speed:10000 Mtu:8901} {Name:ab77f30a26e6dbe MacAddress:12:a0:16:4b:f2:bd Speed:10000 Mtu:8901} {Name:b22e2c211049fc4 MacAddress:52:51:cb:5a:ee:e6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d8:e9:68:c3 Speed:0 Mtu:9001} {Name:br-int MacAddress:1e:69:35:3b:b5:a0 Speed:0 Mtu:8901} {Name:cc01af247e90d4e MacAddress:e6:27:15:30:19:b7 Speed:10000 Mtu:8901} {Name:d0d1b759a87606b MacAddress:66:8a:7c:b3:f0:29 Speed:10000 Mtu:8901} {Name:dd24f9cdcbd495c MacAddress:ca:4e:e8:b5:11:a8 Speed:10000 Mtu:8901} {Name:e7d51ef63b97ea2 MacAddress:b2:48:30:d4:f9:69 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d8:e9:68:c3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2a:cf:98:48:98:96 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:c7:cb:ad:73:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:01:09.227123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227127    5028 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:01:09.227401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227172    5028 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:01:09.227470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227445    5028 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:01:09.227597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227471    5028 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-125.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:01:09.227639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227600    5028 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:01:09.227639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227608    5028 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:01:09.227639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227622    5028 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:01:09.227730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227640    5028 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:01:09.227730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227658    5028 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:01:09.227866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227855    5028 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:01:09.227983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.227974    5028 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:01:09.228021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.228012    5028 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:01:09.228049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.228026    5028 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:01:09.228049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.228037    5028 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:01:09.228049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.228046    5028 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:01:09.228183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.228071    5028 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:01:09.229098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.229079    5028 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:01:09.229098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.229097    5028 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:01:09.229697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.229661    5028 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:01:09.241192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241172    5028 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:01:09.241192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241188    5028 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:01:09.241327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241311    5028 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 22:01:09.241514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241503    5028 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:01:09.241557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241532    5028 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:01:09.241681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241671    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:01:09.241732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241704    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:01:09.241732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241713    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:01:09.241732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241720    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:01:09.241732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241726    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:01:09.241732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241731    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241756    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241762    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241769    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241776    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241785    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241793    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241818    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:01:09.241905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.241823    5028 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:01:09.245450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.245433    5028 server.go:1263] "Started kubelet"
Oct 30 22:01:09.245578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.245532    5028 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:01:09.245943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.245625    5028 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:01:09.245943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.245759    5028 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:01:09.246013 ip-10-0-129-125 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:01:09.246135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.246121    5028 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:01:09.249376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.248500    5028 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.251728    5028 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.251775    5028 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.251798    5028 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:27 +0000 UTC" deadline="2028-07-02 01:36:15.716607745 +0000 UTC"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.251831    5028 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14643h35m6.464779749s"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.251854    5028 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-125.us-west-2.compute.internal\" not found"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.251896    5028 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.251903    5028 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:01:09.252179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.252002    5028 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:01:09.254397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.254373    5028 factory.go:55] Registering systemd factory
Oct 30 22:01:09.254397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.254399    5028 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:01:09.254957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.254929    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be68f52a-8bf4-474e-b28d-133aecbfb229" volumeName="kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.254957    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dc81215-458c-47bb-87e8-ed1893729907" volumeName="kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.254978    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4" volumeName="kubernetes.io/projected/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-kube-api-access-p47rc" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.254990    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be68f52a-8bf4-474e-b28d-133aecbfb229" volumeName="kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255000    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd533e21-2bdf-4973-99f5-568d4a5b0922" volumeName="kubernetes.io/projected/cd533e21-2bdf-4973-99f5-568d4a5b0922-kube-api-access-x7z47" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255007    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d824b6ee-c212-4bea-8c42-5dbd4e57ad60" volumeName="kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-tmp" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255014    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64039614-3fbe-4a3a-b80f-2078ae1cc041" volumeName="kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-config" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255021    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4" volumeName="kubernetes.io/secret/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-serving-cert" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255029    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3282fe-9d41-4c82-9bac-bb74c4209b62" volumeName="kubernetes.io/projected/bd3282fe-9d41-4c82-9bac-bb74c4209b62-kube-api-access-8ptwh" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255037    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be68f52a-8bf4-474e-b28d-133aecbfb229" volumeName="kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-tmp" seLinuxMountContext=""
Oct 30 22:01:09.255040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255045    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63fef56c-376f-4d60-a2c9-c9b02731c5ed" volumeName="kubernetes.io/projected/63fef56c-376f-4d60-a2c9-c9b02731c5ed-kube-api-access-6kqtv" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255052    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255061    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a461f5f1-641c-4f5e-8f29-995036f2f0f7" volumeName="kubernetes.io/configmap/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-config" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255081    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be68f52a-8bf4-474e-b28d-133aecbfb229" volumeName="kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-snapshots" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255090    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be68f52a-8bf4-474e-b28d-133aecbfb229" volumeName="kubernetes.io/projected/be68f52a-8bf4-474e-b28d-133aecbfb229-kube-api-access-slx6l" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255096    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28a3c9a-7726-406c-bfdf-f36065f02a81" volumeName="kubernetes.io/secret/d28a3c9a-7726-406c-bfdf-f36065f02a81-agent-certs" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255103    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d91ca99d-6276-4875-9d6b-e1e49533a930" volumeName="kubernetes.io/projected/d91ca99d-6276-4875-9d6b-e1e49533a930-kube-api-access-92d4g" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255110    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a461f5f1-641c-4f5e-8f29-995036f2f0f7" volumeName="kubernetes.io/projected/a461f5f1-641c-4f5e-8f29-995036f2f0f7-kube-api-access-ld84k" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255117    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0199affc-a6e4-46a9-a96f-351e6bbbafa8" volumeName="kubernetes.io/projected/0199affc-a6e4-46a9-a96f-351e6bbbafa8-kube-api-access-mdcks" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255123    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255130    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10e32bef-cfe9-48eb-84e9-44fdebb43090" volumeName="kubernetes.io/projected/10e32bef-cfe9-48eb-84e9-44fdebb43090-kube-api-access-q4sd9" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255140    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dc81215-458c-47bb-87e8-ed1893729907" volumeName="kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255152    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9" volumeName="kubernetes.io/projected/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-api-access-7pw5s" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255167    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e" volumeName="kubernetes.io/projected/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-kube-api-access-zg4n7" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255174    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255180    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255187    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="914a456e-5637-4208-98bd-ee4078634291" volumeName="kubernetes.io/projected/914a456e-5637-4208-98bd-ee4078634291-kube-api-access-q28g6" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255194    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dc81215-458c-47bb-87e8-ed1893729907" volumeName="kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255203    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165" volumeName="kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255211    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4" volumeName="kubernetes.io/configmap/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-config" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255222    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" volumeName="kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-trusted-ca" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255230    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f572fd28-f04a-45ed-a153-5921471fb276" volumeName="kubernetes.io/configmap/f572fd28-f04a-45ed-a153-5921471fb276-config" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255236    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f572fd28-f04a-45ed-a153-5921471fb276" volumeName="kubernetes.io/secret/f572fd28-f04a-45ed-a153-5921471fb276-serving-cert" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255243    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0199affc-a6e4-46a9-a96f-351e6bbbafa8" volumeName="kubernetes.io/secret/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-key" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255249    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b72179a-878b-4792-939e-6db6499f00bd" volumeName="kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255256    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a790e5-683e-4dfd-82e1-1bde4629e09f" volumeName="kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-stats-auth" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255263    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255274    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="914a456e-5637-4208-98bd-ee4078634291" volumeName="kubernetes.io/secret/914a456e-5637-4208-98bd-ee4078634291-cert" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255281    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" volumeName="kubernetes.io/secret/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-serving-cert" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255289    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a790e5-683e-4dfd-82e1-1bde4629e09f" volumeName="kubernetes.io/projected/56a790e5-683e-4dfd-82e1-1bde4629e09f-kube-api-access-dx9wd" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255300    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d824b6ee-c212-4bea-8c42-5dbd4e57ad60" volumeName="kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-tuned" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255313    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255320    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255328    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9" volumeName="kubernetes.io/empty-dir/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-data-volume" seLinuxMountContext=""
Oct 30 22:01:09.255319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255335    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64039614-3fbe-4a3a-b80f-2078ae1cc041" volumeName="kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255343    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="767b8f6d-306f-45ef-97cb-a5e04e233892" volumeName="kubernetes.io/projected/767b8f6d-306f-45ef-97cb-a5e04e233892-kube-api-access-86rms" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255350    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255360    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f572fd28-f04a-45ed-a153-5921471fb276" volumeName="kubernetes.io/projected/f572fd28-f04a-45ed-a153-5921471fb276-kube-api-access-2qvqc" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255369    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5240e9dc-63a3-436f-af94-ab88f6af0c36" volumeName="kubernetes.io/projected/5240e9dc-63a3-436f-af94-ab88f6af0c36-kube-api-access-5jt6r" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255380    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cafb6fd-d8bd-417e-8b93-d42dbd70d102" volumeName="kubernetes.io/empty-dir/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-tmp-dir" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255387    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cafb6fd-d8bd-417e-8b93-d42dbd70d102" volumeName="kubernetes.io/projected/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-kube-api-access-qhcf9" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255395    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e8b0fdc-6162-4680-b22b-d1eb1656f35b" volumeName="kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255402    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e8b0fdc-6162-4680-b22b-d1eb1656f35b" volumeName="kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-daemon-config" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255411    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a790e5-683e-4dfd-82e1-1bde4629e09f" volumeName="kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-default-certificate" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255419    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28a3c9a-7726-406c-bfdf-f36065f02a81" volumeName="kubernetes.io/configmap/d28a3c9a-7726-406c-bfdf-f36065f02a81-konnectivity-ca" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255426    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a535db2-2694-4965-b4f9-5b478681381e" volumeName="kubernetes.io/projected/1a535db2-2694-4965-b4f9-5b478681381e-kube-api-access-lxt6q" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255438    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf" volumeName="kubernetes.io/projected/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-kube-api-access-mbrzb" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255450    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b72179a-878b-4792-939e-6db6499f00bd" volumeName="kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255459    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b72179a-878b-4792-939e-6db6499f00bd" volumeName="kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255467    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255474    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e8b0fdc-6162-4680-b22b-d1eb1656f35b" volumeName="kubernetes.io/projected/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-kube-api-access-qxhgf" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255481    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64039614-3fbe-4a3a-b80f-2078ae1cc041" volumeName="kubernetes.io/secret/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255488    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e320d2-2d3d-45fb-a44b-41bfe19091ae" volumeName="kubernetes.io/configmap/69e320d2-2d3d-45fb-a44b-41bfe19091ae-nginx-conf" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255495    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" volumeName="kubernetes.io/projected/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-kube-api-access-glgwv" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255502    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f71a60-c64f-4960-934a-6c75e8d94546" volumeName="kubernetes.io/configmap/e7f71a60-c64f-4960-934a-6c75e8d94546-serviceca" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255509    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03d17bd7-a7c6-45b9-b0e5-2d9809543781" volumeName="kubernetes.io/projected/03d17bd7-a7c6-45b9-b0e5-2d9809543781-kube-api-access-n2shr" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255515    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255525    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b72179a-878b-4792-939e-6db6499f00bd" volumeName="kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255535    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64039614-3fbe-4a3a-b80f-2078ae1cc041" volumeName="kubernetes.io/projected/64039614-3fbe-4a3a-b80f-2078ae1cc041-kube-api-access-2kxzk" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255543    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd533e21-2bdf-4973-99f5-568d4a5b0922" volumeName="kubernetes.io/configmap/cd533e21-2bdf-4973-99f5-568d4a5b0922-iptables-alerter-script" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255550    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d824b6ee-c212-4bea-8c42-5dbd4e57ad60" volumeName="kubernetes.io/projected/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-kube-api-access-4rzpr" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255558    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f71a60-c64f-4960-934a-6c75e8d94546" volumeName="kubernetes.io/projected/e7f71a60-c64f-4960-934a-6c75e8d94546-kube-api-access-vwgfj" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255564    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf" volumeName="kubernetes.io/configmap/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-config-volume" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255571    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0199affc-a6e4-46a9-a96f-351e6bbbafa8" volumeName="kubernetes.io/configmap/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-cabundle" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255578    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255586    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dc81215-458c-47bb-87e8-ed1893729907" volumeName="kubernetes.io/projected/1dc81215-458c-47bb-87e8-ed1893729907-kube-api-access-4545h" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255594    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" volumeName="kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255605    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9efd0f0b-f296-4a99-ad21-c3fa3923bb7c" volumeName="kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255615    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" volumeName="kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-config" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255622    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255629    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="051a42ce-76bb-4e7e-a32c-82bff50cd38d" volumeName="kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255636    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be68f52a-8bf4-474e-b28d-133aecbfb229" volumeName="kubernetes.io/secret/be68f52a-8bf4-474e-b28d-133aecbfb229-serving-cert" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255644    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf" volumeName="kubernetes.io/empty-dir/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-tmp-dir" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255651    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43829ec3-a430-4d2e-b2ee-19ae8df72ac9" volumeName="kubernetes.io/projected/43829ec3-a430-4d2e-b2ee-19ae8df72ac9-kube-api-access-kvcgc" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255658    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64039614-3fbe-4a3a-b80f-2078ae1cc041" volumeName="kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-env-overrides" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255666    5028 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e" volumeName="kubernetes.io/configmap/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-telemetry-config" seLinuxMountContext=""
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255676    5028 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255700    5028 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255836    5028 factory.go:153] Registering CRI-O factory
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255851    5028 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255900    5028 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255927    5028 factory.go:103] Registering Raw factory
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.255941    5028 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:01:09.256432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.256318    5028 manager.go:319] Starting recovery of all containers
Oct 30 22:01:09.260054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.260032    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/1.log"
Oct 30 22:01:09.265196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.264872    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/1.log"
Oct 30 22:01:09.281261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.281224    5028 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:01:09.282619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.282595    5028 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:01:09.282619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.282620    5028 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:01:09.282810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.282641    5028 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:01:09.282810 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.282675    5028 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:01:09.299666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.299642    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/1.log"
Oct 30 22:01:09.299791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.299670    5028 generic.go:356] "Generic (PLEG): container finished" podID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" containerID="97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201" exitCode=255
Oct 30 22:01:09.311979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.311948    5028 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="ffab3aab2d0787d1e399a09bdfc52b2908b6ba8debbdb576a8b459b6ab29c481" exitCode=0
Oct 30 22:01:09.311979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.311974    5028 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="7b049ea5d933c7c0a2ee2b802bf62585832fd0c62ccfd94ac745223ccab2b0b8" exitCode=0
Oct 30 22:01:09.311979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.311983    5028 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="9183f53f4211f74f250e74028b3fb9035eff80008ac84060eba0af5933533100" exitCode=0
Oct 30 22:01:09.312247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.311991    5028 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="004f030a10f3b8906ae030b67f556c9597d7efed31644c2dd3c255e584bb22ab" exitCode=0
Oct 30 22:01:09.312247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.311996    5028 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="137d539c1d66fbda26415423aff3d0ed9a2539d5439718a7f3947bc6449db175" exitCode=0
Oct 30 22:01:09.312247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.312001    5028 generic.go:356] "Generic (PLEG): container finished" podID="1dc81215-458c-47bb-87e8-ed1893729907" containerID="d9df2f7d98c19b88b3e6992398ef83c30125210486067270dcc5d9ab468cf79d" exitCode=0
Oct 30 22:01:09.317801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.317771    5028 generic.go:356] "Generic (PLEG): container finished" podID="4b6780e67bf7a04b04eabfaeb0ab1bf0" containerID="009d7ec6a6d6abb742c13ef9b2c1dc93153c61dc96e30a3c47f036ed9eaee2ea" exitCode=0
Oct 30 22:01:09.351965 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.351935    5028 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-125.us-west-2.compute.internal\" not found"
Oct 30 22:01:09.361652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.361632    5028 manager.go:324] Recovery completed
Oct 30 22:01:09.382838 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.382760    5028 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 22:01:09.383232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.383213    5028 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:01:09.384479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.384460    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:01:09.384568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.384493    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:01:09.384568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.384509    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:01:09.387133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387107    5028 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:01:09.387133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387123    5028 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:01:09.387281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387124    5028 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:01:09.387281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387147    5028 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:01:09.387380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387317    5028 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 22:01:09.387380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387333    5028 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 22:01:09.387380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387358    5028 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 22:01:09.387380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387367    5028 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 22:01:09.387380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387375    5028 policy_none.go:50] "Start"
Oct 30 22:01:09.387588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387388    5028 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:01:09.387588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.387401    5028 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:01:09.388039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.388024    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:01:09.388121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.388051    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:01:09.388121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.388068    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:01:09.399716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.399671    5028 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 22:01:09.399716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.399716    5028 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 22:01:09.399873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.399726    5028 policy_none.go:44] "Start"
Oct 30 22:01:09.404775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.404741    5028 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:01:09.410302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.410290    5028 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:01:09.410384 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.410315    5028 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:01:09.410384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.410326    5028 server.go:87] "Starting device plugin registration server"
Oct 30 22:01:09.411009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.410994    5028 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:01:09.411046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.411012    5028 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:01:09.412020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.411160    5028 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:01:09.412020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.411256    5028 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:01:09.412020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.411293    5028 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:01:09.412020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.411299    5028 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:01:09.412020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.411345    5028 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:01:09.411279415Z","UUID":"dfa14bec-81fb-4c0f-9020-2b8680469525","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:01:09.412820 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.412801    5028 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:01:09.412908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.412828    5028 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:01:09.412908 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.412844    5028 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-125.us-west-2.compute.internal\" not found"
Oct 30 22:01:09.412908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.412849    5028 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:01:09.431293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.431273    5028 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 22:01:09.431989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.431976    5028 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:01:09.438140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.438124    5028 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:01:09.438270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.438254    5028 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:01:09.486000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.485972    5028 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:01:09.511235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.511210    5028 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:01:09.512318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.512296    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:01:09.512412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.512329    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:01:09.512412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.512343    5028 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:01:09.512412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.512396    5028 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.583064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.582982    5028 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"]
Oct 30 22:01:09.584187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584163    5028 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.584548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584491    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"4b6780e67bf7a04b04eabfaeb0ab1bf0","Type":"ContainerStarted","Data":"6997cdf5a157ebe8e29a1827200605210a7f5da237cbdbea9f557b63569c747a"}
Oct 30 22:01:09.584668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584649    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"4b6780e67bf7a04b04eabfaeb0ab1bf0","Type":"ContainerDied","Data":"009d7ec6a6d6abb742c13ef9b2c1dc93153c61dc96e30a3c47f036ed9eaee2ea"}
Oct 30 22:01:09.584744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584706    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"4b6780e67bf7a04b04eabfaeb0ab1bf0","Type":"ContainerStarted","Data":"79974435d422eb2403f3b7f1c275a7d75040204c4e85cbebbbebc5b3d571812b"}
Oct 30 22:01:09.584789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584776    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"95227d9216a597ca813ef85310eb7d77","Type":"ContainerStarted","Data":"a1cb7153bc26da97abfdb9450902be796da1e5bb61ea5e0a2ff9c1d00f4c1fc0"}
Oct 30 22:01:09.584841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584802    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal" event={"ID":"95227d9216a597ca813ef85310eb7d77","Type":"ContainerStarted","Data":"150b4419df90e46dbb5784b02aa9a5e901b94a71408943e551f34575b6710be5"}
Oct 30 22:01:09.584841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.584830    5028 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.617938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.617908    5028 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.618067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.617982    5028 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.664212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.664098    5028 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:01:09.664212 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.664186    5028 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.665853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.665835    5028 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:01:09.665930 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:09.665877    5028 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.732321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.732283    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.732488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.732328    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.732488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.732362    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95227d9216a597ca813ef85310eb7d77-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"95227d9216a597ca813ef85310eb7d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.832740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.832681    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95227d9216a597ca813ef85310eb7d77-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"95227d9216a597ca813ef85310eb7d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.832892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.832753    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.832892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.832788    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.832892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.832796    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.832892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.832760    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/95227d9216a597ca813ef85310eb7d77-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"95227d9216a597ca813ef85310eb7d77\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:09.832892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:09.832828    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b6780e67bf7a04b04eabfaeb0ab1bf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" (UID: \"4b6780e67bf7a04b04eabfaeb0ab1bf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:10.228433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.228402    5028 apiserver.go:52] "Watching apiserver"
Oct 30 22:01:10.355746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.355715    5028 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:10.355914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.355816    5028 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:10.374832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.374800    5028 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:01:10.375532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.375502    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-pbcdv","openshift-ingress-canary/ingress-canary-s96ng","kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal","openshift-image-registry/image-registry-876dbf8dd-czb2r","openshift-network-diagnostics/network-check-target-lm2gp","kube-system/kas-connection-checker-685895c7c9-cgm7l","openshift-insights/insights-runtime-extractor-nxvq5","openshift-ovn-kubernetes/ovnkube-node-tw7xp","openshift-service-ca/service-ca-5fffc97bdf-jt4tn","openshift-image-registry/image-registry-6d45ff7cc7-twst9","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh","kube-system/kas-connection-checker-685895c7c9-csjz8","openshift-console-operator/console-operator-7cd56b8dbc-q2lps","openshift-network-console/networking-console-plugin-86c478bb88-g99mq","openshift-ingress/router-default-86b69957bf-6kffb","openshift-insights/periodic-gathering-slw2d-b5qs2","openshift-multus/multus-m48h8","openshift-dns/node-resolver-jg2sh","openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6","openshift-cluster-node-tuning-operator/tuned-b4q4p","openshift-multus/multus-additional-cni-plugins-fp9vl","openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs","kube-system/global-pull-secret-syncer-rbvwt","openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv","openshift-image-registry/node-ca-gpmjd","openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86","openshift-dns/dns-default-zzj6z","kube-system/kas-connection-checker-685895c7c9-7dn4w","openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf","openshift-multus/network-metrics-daemon-hfs8l","kube-system/konnectivity-agent-4j6gf","openshift-insights/insights-operator-7c4986d449-cxs8w","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"]
Oct 30 22:01:10.377501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.377479    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.377765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.377748    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.379268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.379247    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.380728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.380711    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.381731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.381711    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:10.388282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.388254    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:10.390749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.390734    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.391222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.391210    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.392339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.392323    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.413761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.413734    5028 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:01:10.413888 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.413789    5028 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:10.414368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.414348    5028 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:01:10.414451 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.414388    5028 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.us-west-2.compute.internal"
Oct 30 22:01:10.427113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.427086    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.427440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.427422    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.428736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428722    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:01:10.428839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428821    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:01:10.428907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428876    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.428907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428893    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:01:10.429008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428916    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:01:10.429008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428924    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.429008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428916    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:01:10.429008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428939    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.429008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428978    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.429008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.428978    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.429172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.429112    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:01:10.430390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430346    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:01:10.430854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430597    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:01:10.430854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430745    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-22kdc\""
Oct 30 22:01:10.430854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430818    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:01:10.430854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430844    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hpnbl\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430849    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wfrf6\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430879    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.430973    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s28xf\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.431027    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.431039    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gg6vq\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.431245    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fskfb\""
Oct 30 22:01:10.431299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.431255    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wr55q\""
Oct 30 22:01:10.432019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.431992    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:01:10.432167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432033    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.432295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432265    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:01:10.432295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432286    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.432459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432403    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:01:10.432459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432453    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.432627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432485    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.432627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432511    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:01:10.432627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432521    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.432627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432564    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.433001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432742    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.433001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432764    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.433001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.432840    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.433244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.433229    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x2z2x\""
Oct 30 22:01:10.454935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.454881    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:10.455561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.455545    5028 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:01:10.476511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.476456    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:01:10.476677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.476658    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:01:10.477989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.477963    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:01:10.478121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.477993    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.478121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.477975    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:01:10.478121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478117    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\""
Oct 30 22:01:10.478372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478355    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hgnsb\""
Oct 30 22:01:10.478493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478404    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:01:10.478493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478437    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:01:10.478493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478490    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:01:10.478704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478676    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:01:10.478765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478711    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.478765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478711    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.478765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478736    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.478765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478725    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:01:10.478765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478756    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.478977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.478965    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.479902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.479889    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.479967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.479914    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.480042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.480029    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cllmb\""
Oct 30 22:01:10.480123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.480037    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.480123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.480049    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdfn5\""
Oct 30 22:01:10.480635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.480609    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.480635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.480620    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:01:10.480799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.480664    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:01:10.481106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481071    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:01:10.481267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481155    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8d2dc\""
Oct 30 22:01:10.481267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481218    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj92j\""
Oct 30 22:01:10.481267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481228    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g5t42\""
Oct 30 22:01:10.481461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481321    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rstdk\""
Oct 30 22:01:10.481461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481345    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:01:10.481461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481371    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:01:10.481461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481412    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.481461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481432    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481490    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481512    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481555    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481595    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481630    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481657    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481667    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.481741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481731    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481752    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481763    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481774    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481812    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481822    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481835    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481850    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481863    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.481870    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482008    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482043    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482010    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqg6l\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482051    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482106    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfj4m\""
Oct 30 22:01:10.482183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482155    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.482926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482195    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:01:10.482926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482297    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6qlsl\""
Oct 30 22:01:10.483026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.482940    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd25c\""
Oct 30 22:01:10.483026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.483019    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xd6xx\""
Oct 30 22:01:10.483119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.483055    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75hcr\""
Oct 30 22:01:10.483194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.483175    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:01:10.483404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.483387    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:01:10.484498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.484476    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:01:10.484908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.484891    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9mpbc\""
Oct 30 22:01:10.485240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.485225    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skckn\""
Oct 30 22:01:10.485599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.485584    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlw52\""
Oct 30 22:01:10.485814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.485800    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.486316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.486299    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:01:10.487370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.487351    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6l5qx\""
Oct 30 22:01:10.487535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.487522    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:01:10.487714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.487700    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzw7r\""
Oct 30 22:01:10.487812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.487795    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\""
Oct 30 22:01:10.488794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.488776    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:01:10.536789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.536728    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-k8s-cni-cncf-io\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.537152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.537130    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-netns\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.537427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.537387    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-kubelet\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.537593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.537579    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-conf\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.538090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538058    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-lib-modules\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.538226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538106    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-log-socket\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.538226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538139    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-env-overrides\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.538226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538178    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f71a60-c64f-4960-934a-6c75e8d94546-host\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.538226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538210    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f71a60-c64f-4960-934a-6c75e8d94546-serviceca\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.538530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538260    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-serving-cert\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:01:10.538530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538454    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7f71a60-c64f-4960-934a-6c75e8d94546-serviceca\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.538530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538483    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-serving-cert\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:01:10.538530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538522    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysconfig\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.538749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538551    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-config-volume\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.538749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538576    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47rc\" (UniqueName: \"kubernetes.io/projected/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-kube-api-access-p47rc\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:01:10.538749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538598    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.538749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538623    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kqtv\" (UniqueName: \"kubernetes.io/projected/63fef56c-376f-4d60-a2c9-c9b02731c5ed-kube-api-access-6kqtv\") pod \"network-check-source-6f78d59f99-zl5rs\" (UID: \"63fef56c-376f-4d60-a2c9-c9b02731c5ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:01:10.538749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538666    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.539239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538819    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-env-overrides\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.539239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.538964    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.539239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539130    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcks\" (UniqueName: \"kubernetes.io/projected/0199affc-a6e4-46a9-a96f-351e6bbbafa8-kube-api-access-mdcks\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:10.539239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539168    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrzb\" (UniqueName: \"kubernetes.io/projected/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-kube-api-access-mbrzb\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.539239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539194    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jt6r\" (UniqueName: \"kubernetes.io/projected/5240e9dc-63a3-436f-af94-ab88f6af0c36-kube-api-access-5jt6r\") pod \"kas-connection-checker-685895c7c9-csjz8\" (UID: \"5240e9dc-63a3-436f-af94-ab88f6af0c36\") " pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:01:10.539239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539221    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-hosts-file\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.539490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539247    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovn-node-metrics-cert\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.539490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539271    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-default-certificate\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.539490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539296    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-socket-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.539490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539318    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28a3c9a-7726-406c-bfdf-f36065f02a81-agent-certs\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:01:10.539490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539339    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cni-binary-copy\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.539737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539654    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-default-certificate\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.539737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539679    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cni-binary-copy\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.539883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539862    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovn-node-metrics-cert\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.539966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539955    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-config-volume\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.540025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.539998    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld84k\" (UniqueName: \"kubernetes.io/projected/a461f5f1-641c-4f5e-8f29-995036f2f0f7-kube-api-access-ld84k\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.540084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540026    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-bin\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.540084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540014    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d28a3c9a-7726-406c-bfdf-f36065f02a81-agent-certs\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:01:10.540084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540058    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd533e21-2bdf-4973-99f5-568d4a5b0922-host-slash\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.540250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540122    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-config\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:01:10.540250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540162    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.540250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540177    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.540250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540217    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pw5s\" (UniqueName: \"kubernetes.io/projected/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-api-access-7pw5s\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.540250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540242    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f572fd28-f04a-45ed-a153-5921471fb276-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:01:10.540485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540268    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgfj\" (UniqueName: \"kubernetes.io/projected/e7f71a60-c64f-4960-934a-6c75e8d94546-kube-api-access-vwgfj\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.540485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540293    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3c9a-7726-406c-bfdf-f36065f02a81-konnectivity-ca\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:01:10.540485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540364    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.540485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540475    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3c9a-7726-406c-bfdf-f36065f02a81-konnectivity-ca\") pod \"konnectivity-agent-4j6gf\" (UID: \"d28a3c9a-7726-406c-bfdf-f36065f02a81\") " pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:01:10.540701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540510    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-hostroot\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.540701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540537    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/914a456e-5637-4208-98bd-ee4078634291-cert\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:01:10.540701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540565    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9wd\" (UniqueName: \"kubernetes.io/projected/56a790e5-683e-4dfd-82e1-1bde4629e09f-kube-api-access-dx9wd\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.540701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540627    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-modprobe-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.540701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540672    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540709    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-systemd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540739    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540750    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-config\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540762    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-sys-fs\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540786    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-key\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540851    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-cabundle\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540880    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-var-lib-kubelet\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540888    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f572fd28-f04a-45ed-a153-5921471fb276-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540896    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-netns\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.540931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540916    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-cnibin\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540934    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4545h\" (UniqueName: \"kubernetes.io/projected/1dc81215-458c-47bb-87e8-ed1893729907-kube-api-access-4545h\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540968    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-snapshots\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.540984    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-daemon-config\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541001    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-run\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541016    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-sys\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541034    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-netd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541055    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541125    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541160    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541177    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-snapshots\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541181    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f572fd28-f04a-45ed-a153-5921471fb276-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541231    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69e320d2-2d3d-45fb-a44b-41bfe19091ae-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541262    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcf9\" (UniqueName: \"kubernetes.io/projected/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-kube-api-access-qhcf9\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541293    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slx6l\" (UniqueName: \"kubernetes.io/projected/be68f52a-8bf4-474e-b28d-133aecbfb229-kube-api-access-slx6l\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541299    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f572fd28-f04a-45ed-a153-5921471fb276-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541343    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.541355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541364    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541401    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541427    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-bin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541454    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-script-lib\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541480    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxt6q\" (UniqueName: \"kubernetes.io/projected/1a535db2-2694-4965-b4f9-5b478681381e-kube-api-access-lxt6q\") pod \"kas-connection-checker-685895c7c9-cgm7l\" (UID: \"1a535db2-2694-4965-b4f9-5b478681381e\") " pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541508    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-multus-certs\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541534    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541560    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541586    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-etc-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541663    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-daemon-config\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541725    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-service-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541755    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-system-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541778    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-multus\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541803    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541816    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/69e320d2-2d3d-45fb-a44b-41bfe19091ae-nginx-conf\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541829    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2shr\" (UniqueName: \"kubernetes.io/projected/03d17bd7-a7c6-45b9-b0e5-2d9809543781-kube-api-access-n2shr\") pod \"kas-connection-checker-685895c7c9-7dn4w\" (UID: \"03d17bd7-a7c6-45b9-b0e5-2d9809543781\") " pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541860    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541887    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-device-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541914    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541936    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-dbus\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541961    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-tmp-dir\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.541986    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542013    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-data-volume\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542038    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzpr\" (UniqueName: \"kubernetes.io/projected/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-kube-api-access-4rzpr\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542064    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542095    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cnibin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.542104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542119    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-conf-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542146    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542155    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-key\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542173    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxzk\" (UniqueName: \"kubernetes.io/projected/64039614-3fbe-4a3a-b80f-2078ae1cc041-kube-api-access-2kxzk\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542197    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542223    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542251    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-config\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542276    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542305    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-host\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542589    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542791    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542823    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542965    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0199affc-a6e4-46a9-a96f-351e6bbbafa8-signing-cabundle\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.542592    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-slash\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543069    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543074    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.543211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543192    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543268    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92d4g\" (UniqueName: \"kubernetes.io/projected/d91ca99d-6276-4875-9d6b-e1e49533a930-kube-api-access-92d4g\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543310    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543327    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/914a456e-5637-4208-98bd-ee4078634291-cert\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543341    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-socket-dir-parent\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543371    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-trusted-ca\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543381    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543400    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgwv\" (UniqueName: \"kubernetes.io/projected/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-kube-api-access-glgwv\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543410    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543428    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptwh\" (UniqueName: \"kubernetes.io/projected/bd3282fe-9d41-4c82-9bac-bb74c4209b62-kube-api-access-8ptwh\") pod \"migrator-66df5d496b-x6s86\" (UID: \"bd3282fe-9d41-4c82-9bac-bb74c4209b62\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543457    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543484    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4sd9\" (UniqueName: \"kubernetes.io/projected/10e32bef-cfe9-48eb-84e9-44fdebb43090-kube-api-access-q4sd9\") pod \"volume-data-source-validator-58c9f5f46d-hsdm7\" (UID: \"10e32bef-cfe9-48eb-84e9-44fdebb43090\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543514    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543518    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-config\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543627    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543637    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-tmp-dir\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543716    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-data-volume\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543612    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.543942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543816    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be68f52a-8bf4-474e-b28d-133aecbfb229-service-ca-bundle\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.543976    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64039614-3fbe-4a3a-b80f-2078ae1cc041-ovnkube-script-lib\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544015    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-original-pull-secret\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544172    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28g6\" (UniqueName: \"kubernetes.io/projected/914a456e-5637-4208-98bd-ee4078634291-kube-api-access-q28g6\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544263    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-systemd\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544298    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-node-log\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544330    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-trusted-ca\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544354    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-kubelet\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544359    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544400    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-tmp-dir\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544441    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4n7\" (UniqueName: \"kubernetes.io/projected/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-kube-api-access-zg4n7\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544491    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rms\" (UniqueName: \"kubernetes.io/projected/767b8f6d-306f-45ef-97cb-a5e04e233892-kube-api-access-86rms\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544514    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544522    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-tmp-dir\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544542    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-serving-cert\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544653    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-tmp\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.544735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544727    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544771    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-tmp\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544774    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-systemd-units\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544818    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-serving-cert\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544858    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-var-lib-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544904    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544941    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-crio-socket\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.544987    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-config\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545031    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545147    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545190    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545225    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhgf\" (UniqueName: \"kubernetes.io/projected/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-kube-api-access-qxhgf\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545252    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-config\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.545324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545306    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be68f52a-8bf4-474e-b28d-133aecbfb229-serving-cert\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545372    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545384    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcgc\" (UniqueName: \"kubernetes.io/projected/43829ec3-a430-4d2e-b2ee-19ae8df72ac9-kube-api-access-kvcgc\") pod \"migrator-66df5d496b-n8qq6\" (UID: \"43829ec3-a430-4d2e-b2ee-19ae8df72ac9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545543    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545547    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be68f52a-8bf4-474e-b28d-133aecbfb229-serving-cert\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545602    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvqc\" (UniqueName: \"kubernetes.io/projected/f572fd28-f04a-45ed-a153-5921471fb276-kube-api-access-2qvqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545670    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545717    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545762    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545851    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.545874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545870    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.546169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545923    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-stats-auth\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.546169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.545965    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.546169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546063    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-registration-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.546169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546115    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.546169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546160    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-stats-auth\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.546306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546291    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.546336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546319    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.546380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546364    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.546429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546413    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-tuned\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.546482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546462    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-kubelet-config\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.546521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546508    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd533e21-2bdf-4973-99f5-568d4a5b0922-iptables-alerter-script\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.546568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546533    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7z47\" (UniqueName: \"kubernetes.io/projected/cd533e21-2bdf-4973-99f5-568d4a5b0922-kube-api-access-x7z47\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.546568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546539    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-tuned\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.546665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546572    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.546665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546581    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.546665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546595    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.546665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546636    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-etc-kubernetes\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.546846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546718    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.547752    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-tmp\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.547787    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.547812    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-kubernetes\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.547836    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-ovn\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.547858    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.548049    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546789    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cd533e21-2bdf-4973-99f5-568d4a5b0922-iptables-alerter-script\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.548154    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be68f52a-8bf4-474e-b28d-133aecbfb229-tmp\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.549646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.546819    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dc81215-458c-47bb-87e8-ed1893729907-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.597132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.597094    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kqtv\" (UniqueName: \"kubernetes.io/projected/63fef56c-376f-4d60-a2c9-c9b02731c5ed-kube-api-access-6kqtv\") pod \"network-check-source-6f78d59f99-zl5rs\" (UID: \"63fef56c-376f-4d60-a2c9-c9b02731c5ed\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zl5rs"
Oct 30 22:01:10.598219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.597606    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jt6r\" (UniqueName: \"kubernetes.io/projected/5240e9dc-63a3-436f-af94-ab88f6af0c36-kube-api-access-5jt6r\") pod \"kas-connection-checker-685895c7c9-csjz8\" (UID: \"5240e9dc-63a3-436f-af94-ab88f6af0c36\") " pod="kube-system/kas-connection-checker-685895c7c9-csjz8"
Oct 30 22:01:10.598219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.597612    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrzb\" (UniqueName: \"kubernetes.io/projected/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-kube-api-access-mbrzb\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.598219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.597829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9wd\" (UniqueName: \"kubernetes.io/projected/56a790e5-683e-4dfd-82e1-1bde4629e09f-kube-api-access-dx9wd\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.598219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.598175    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47rc\" (UniqueName: \"kubernetes.io/projected/b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4-kube-api-access-p47rc\") pod \"service-ca-operator-544d749c8c-m592w\" (UID: \"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w"
Oct 30 22:01:10.598494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.598470    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld84k\" (UniqueName: \"kubernetes.io/projected/a461f5f1-641c-4f5e-8f29-995036f2f0f7-kube-api-access-ld84k\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.601762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.601715    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.612852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.612815    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slx6l\" (UniqueName: \"kubernetes.io/projected/be68f52a-8bf4-474e-b28d-133aecbfb229-kube-api-access-slx6l\") pod \"insights-operator-7c4986d449-cxs8w\" (UID: \"be68f52a-8bf4-474e-b28d-133aecbfb229\") " pod="openshift-insights/insights-operator-7c4986d449-cxs8w"
Oct 30 22:01:10.613048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.613030    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcks\" (UniqueName: \"kubernetes.io/projected/0199affc-a6e4-46a9-a96f-351e6bbbafa8-kube-api-access-mdcks\") pod \"service-ca-5fffc97bdf-jt4tn\" (UID: \"0199affc-a6e4-46a9-a96f-351e6bbbafa8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jt4tn"
Oct 30 22:01:10.613320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.613290    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcf9\" (UniqueName: \"kubernetes.io/projected/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-kube-api-access-qhcf9\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.613502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.613486    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgfj\" (UniqueName: \"kubernetes.io/projected/e7f71a60-c64f-4960-934a-6c75e8d94546-kube-api-access-vwgfj\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.615677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.615658    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pw5s\" (UniqueName: \"kubernetes.io/projected/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-api-access-7pw5s\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.617983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.617958    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92d4g\" (UniqueName: \"kubernetes.io/projected/d91ca99d-6276-4875-9d6b-e1e49533a930-kube-api-access-92d4g\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:10.618063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.617962    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") pod \"periodic-gathering-slw2d-b5qs2\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") " pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:01:10.618158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.618142    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4sd9\" (UniqueName: \"kubernetes.io/projected/10e32bef-cfe9-48eb-84e9-44fdebb43090-kube-api-access-q4sd9\") pod \"volume-data-source-validator-58c9f5f46d-hsdm7\" (UID: \"10e32bef-cfe9-48eb-84e9-44fdebb43090\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hsdm7"
Oct 30 22:01:10.618248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.618233    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptwh\" (UniqueName: \"kubernetes.io/projected/bd3282fe-9d41-4c82-9bac-bb74c4209b62-kube-api-access-8ptwh\") pod \"migrator-66df5d496b-x6s86\" (UID: \"bd3282fe-9d41-4c82-9bac-bb74c4209b62\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86"
Oct 30 22:01:10.619588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.619573    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxzk\" (UniqueName: \"kubernetes.io/projected/64039614-3fbe-4a3a-b80f-2078ae1cc041-kube-api-access-2kxzk\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.619667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.619652    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.619726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.619662    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgwv\" (UniqueName: \"kubernetes.io/projected/e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b-kube-api-access-glgwv\") pod \"console-operator-7cd56b8dbc-q2lps\" (UID: \"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:10.620764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.620748    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2shr\" (UniqueName: \"kubernetes.io/projected/03d17bd7-a7c6-45b9-b0e5-2d9809543781-kube-api-access-n2shr\") pod \"kas-connection-checker-685895c7c9-7dn4w\" (UID: \"03d17bd7-a7c6-45b9-b0e5-2d9809543781\") " pod="kube-system/kas-connection-checker-685895c7c9-7dn4w"
Oct 30 22:01:10.622137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.622121    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzpr\" (UniqueName: \"kubernetes.io/projected/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-kube-api-access-4rzpr\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.622835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.622815    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4545h\" (UniqueName: \"kubernetes.io/projected/1dc81215-458c-47bb-87e8-ed1893729907-kube-api-access-4545h\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.623696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.623665    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.623806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.623786    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7z47\" (UniqueName: \"kubernetes.io/projected/cd533e21-2bdf-4973-99f5-568d4a5b0922-kube-api-access-x7z47\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.623860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.623818    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcgc\" (UniqueName: \"kubernetes.io/projected/43829ec3-a430-4d2e-b2ee-19ae8df72ac9-kube-api-access-kvcgc\") pod \"migrator-66df5d496b-n8qq6\" (UID: \"43829ec3-a430-4d2e-b2ee-19ae8df72ac9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6"
Oct 30 22:01:10.624414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.624393    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvqc\" (UniqueName: \"kubernetes.io/projected/f572fd28-f04a-45ed-a153-5921471fb276-kube-api-access-2qvqc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d6swh\" (UID: \"f572fd28-f04a-45ed-a153-5921471fb276\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh"
Oct 30 22:01:10.624714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.624677    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfccn\" (UniqueName: \"kubernetes.io/projected/9efd0f0b-f296-4a99-ad21-c3fa3923bb7c-kube-api-access-cfccn\") pod \"network-check-target-lm2gp\" (UID: \"9efd0f0b-f296-4a99-ad21-c3fa3923bb7c\") " pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:10.624920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.624905    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4n7\" (UniqueName: \"kubernetes.io/projected/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-kube-api-access-zg4n7\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.625015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.625001    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxt6q\" (UniqueName: \"kubernetes.io/projected/1a535db2-2694-4965-b4f9-5b478681381e-kube-api-access-lxt6q\") pod \"kas-connection-checker-685895c7c9-cgm7l\" (UID: \"1a535db2-2694-4965-b4f9-5b478681381e\") " pod="kube-system/kas-connection-checker-685895c7c9-cgm7l"
Oct 30 22:01:10.625979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.625961    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.628029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.628011    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28g6\" (UniqueName: \"kubernetes.io/projected/914a456e-5637-4208-98bd-ee4078634291-kube-api-access-q28g6\") pod \"ingress-canary-s96ng\" (UID: \"914a456e-5637-4208-98bd-ee4078634291\") " pod="openshift-ingress-canary/ingress-canary-s96ng"
Oct 30 22:01:10.628907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.628889    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhgf\" (UniqueName: \"kubernetes.io/projected/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-kube-api-access-qxhgf\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.628960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.628913    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rms\" (UniqueName: \"kubernetes.io/projected/767b8f6d-306f-45ef-97cb-a5e04e233892-kube-api-access-86rms\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.652148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652105    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:10.652293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652154    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.652293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652212    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.652293 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.652277    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:10.652421 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.652299    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:10.652421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652337    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.652421 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.652367    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.152351191 +0000 UTC m=+2.001864161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:10.652421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652276    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.652421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652399    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-registration-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.652421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652420    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652445    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652483    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-kubelet-config\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652517    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652543    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-etc-kubernetes\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652568    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652593    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-kubernetes\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652590    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652616    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-ovn\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652624    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-etc-kubernetes\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652633    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-kubelet-config\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.652711    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.152677143 +0000 UTC m=+2.002190111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : configmap references non-existent config key: service-ca.crt
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652709    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-kubernetes\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.652727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652719    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-ovn\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652747    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-k8s-cni-cncf-io\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652763    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-netns\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.652768    5028 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652552    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-registration-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652804    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-kubelet\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652824    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-k8s-cni-cncf-io\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.652779    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-kubelet\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.652888    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.152870928 +0000 UTC m=+2.002383896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:10.653255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653106    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.653785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653765    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-netns\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.653831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653809    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-conf\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.653831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653825    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-lib-modules\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.653912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653839    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-log-socket\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.653912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653855    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f71a60-c64f-4960-934a-6c75e8d94546-host\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.653912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653870    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysconfig\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.653912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653889    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653934    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653939    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysconfig\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653943    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-log-socket\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653965    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-hosts-file\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653968    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-lib-modules\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653983    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-sysctl-conf\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654003    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cafb6fd-d8bd-417e-8b93-d42dbd70d102-hosts-file\") pod \"node-resolver-jg2sh\" (UID: \"4cafb6fd-d8bd-417e-8b93-d42dbd70d102\") " pod="openshift-dns/node-resolver-jg2sh"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653990    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-socket-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654024    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-socket-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654036    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-bin\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.653987    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7f71a60-c64f-4960-934a-6c75e8d94546-host\") pod \"node-ca-gpmjd\" (UID: \"e7f71a60-c64f-4960-934a-6c75e8d94546\") " pod="openshift-image-registry/node-ca-gpmjd"
Oct 30 22:01:10.654063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654059    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd533e21-2bdf-4973-99f5-568d4a5b0922-host-slash\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654083    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-bin\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654085    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-hostroot\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654103    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-hostroot\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654109    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-modprobe-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654125    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-systemd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654130    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cd533e21-2bdf-4973-99f5-568d4a5b0922-host-slash\") pod \"iptables-alerter-pbcdv\" (UID: \"cd533e21-2bdf-4973-99f5-568d4a5b0922\") " pod="openshift-network-operator/iptables-alerter-pbcdv"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654162    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-systemd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654168    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-sys-fs\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654197    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-modprobe-d\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654204    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-var-lib-kubelet\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654229    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-netns\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654234    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-sys-fs\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654236    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-var-lib-kubelet\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654262    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-cnibin\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654287    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-run\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654307    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-sys\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654328    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-netd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654356    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-cni-netd\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654329    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-cnibin\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654356    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-run\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654354    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654386    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-sys\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.654393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654327    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-run-netns\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654400    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654428    5028 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654434    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654436    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-bin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654457    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-bin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654456    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-multus-certs\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654483    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.154467987 +0000 UTC m=+2.003980941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654485    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-run-multus-certs\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654500    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654526    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-etc-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654550    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-system-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654566    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-etc-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654571    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-multus\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654585    5028 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654597    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-system-cni-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654600    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654625    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654639    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-host-var-lib-cni-multus\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654629    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-device-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654673    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-device-dir\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654705    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.154675106 +0000 UTC m=+2.004188056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654726    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654752    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-dbus\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654779    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cnibin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654804    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/767b8f6d-306f-45ef-97cb-a5e04e233892-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxwzf\" (UID: \"767b8f6d-306f-45ef-97cb-a5e04e233892\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxwzf"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654814    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165-dbus\") pod \"global-pull-secret-syncer-rbvwt\" (UID: \"8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165\") " pod="kube-system/global-pull-secret-syncer-rbvwt"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654802    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-conf-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654838    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654844    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-cnibin\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654846    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-conf-dir\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654854    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-host\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654868    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-slash\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654875    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-host\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654885    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654897    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-slash\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654907    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-socket-dir-parent\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654910    5028 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654948    5028 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654957    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.154946227 +0000 UTC m=+2.004459181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : secret "metrics-daemon-secret" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654955    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e8b0fdc-6162-4680-b22b-d1eb1656f35b-multus-socket-dir-parent\") pod \"multus-m48h8\" (UID: \"4e8b0fdc-6162-4680-b22b-d1eb1656f35b\") " pod="openshift-multus/multus-m48h8"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654978    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.654993    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.15497973 +0000 UTC m=+2.004492689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.654946    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-run-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655021    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655038    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-systemd\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655060    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-node-log\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655074    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-kubelet\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655091    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655110    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-node-log\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655113    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655121    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-host-kubelet\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.655104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655137    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-systemd-units\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655150    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d824b6ee-c212-4bea-8c42-5dbd4e57ad60-etc-systemd\") pod \"tuned-b4q4p\" (UID: \"d824b6ee-c212-4bea-8c42-5dbd4e57ad60\") " pod="openshift-cluster-node-tuning-operator/tuned-b4q4p"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655161    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-var-lib-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655167    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-systemd-units\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655091    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655185    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655190    5028 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655206    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-crio-socket\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655225    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.155217415 +0000 UTC m=+2.004730420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655190    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64039614-3fbe-4a3a-b80f-2078ae1cc041-var-lib-openvswitch\") pod \"ovnkube-node-tw7xp\" (UID: \"64039614-3fbe-4a3a-b80f-2078ae1cc041\") " pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655191    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655279    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-crio-socket\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655288    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.155273319 +0000 UTC m=+2.004786277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655174    5028 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.655310    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dc81215-458c-47bb-87e8-ed1893729907-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9vl\" (UID: \"1dc81215-458c-47bb-87e8-ed1893729907\") " pod="openshift-multus/multus-additional-cni-plugins-fp9vl"
Oct 30 22:01:10.656560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:10.655324    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.155315612 +0000 UTC m=+2.004828577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:10.680512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.680488    5028 scope.go:122] "RemoveContainer" containerID="97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201"
Oct 30 22:01:10.695630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:10.695453    5028 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:01:11.026289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.026201    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:11.159400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159363    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159405    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159448    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159475    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159504    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159518    5028 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159534    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159558    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:11.159593 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159578    5028 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159616    5028 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159584    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159566303 +0000 UTC m=+3.009079274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159647    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159653    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159641089 +0000 UTC m=+3.009154039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159620    5028 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159661    5028 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159666    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159660558 +0000 UTC m=+3.009173508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : secret "metrics-daemon-secret" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159704    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159676658 +0000 UTC m=+3.009189614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159659    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159723    5028 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159724    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159713125 +0000 UTC m=+3.009226082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159767    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159801    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.159831    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159840    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159825103 +0000 UTC m=+3.009338069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159846    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159858    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159870    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls podName:2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159858036 +0000 UTC m=+3.009370999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nxvq5" (UID: "2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159886    5028 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159894    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159885016 +0000 UTC m=+3.009397977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:11.160028 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.159920    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.159910861 +0000 UTC m=+3.009423813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:11.161007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.160413    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56a790e5-683e-4dfd-82e1-1bde4629e09f-service-ca-bundle\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:11.211053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.211022    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:11.364122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.364028    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:01:11.364522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.364475    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/1.log"
Oct 30 22:01:11.364522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.364503    5028 generic.go:356] "Generic (PLEG): container finished" podID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b" containerID="773bce85c4f07b008782382cd84840b2ad511c01786eca90cebcb89c37344991" exitCode=255
Oct 30 22:01:11.364606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.364573    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" event={"ID":"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b","Type":"ContainerDied","Data":"773bce85c4f07b008782382cd84840b2ad511c01786eca90cebcb89c37344991"}
Oct 30 22:01:11.364673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.364659    5028 scope.go:122] "RemoveContainer" containerID="97de41bb6defdf29153c331b20cefb961e37f39b09d471fbd39e3f7d1de99201"
Oct 30 22:01:11.365137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.365119    5028 scope.go:122] "RemoveContainer" containerID="773bce85c4f07b008782382cd84840b2ad511c01786eca90cebcb89c37344991"
Oct 30 22:01:11.365322 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:11.365298    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q2lps_openshift-console-operator(e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" podUID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b"
Oct 30 22:01:11.599900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.599873    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jg2sh_4cafb6fd-d8bd-417e-8b93-d42dbd70d102/dns-node-resolver/0.log"
Oct 30 22:01:11.677601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.677527    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:11.825385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:11.825337    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.169977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.169942    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:12.170176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.169994    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:12.170176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170024    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:12.170176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170052    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:12.170176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170100    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:12.170386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170182    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:12.170386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170216    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:12.170386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170260    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:12.170386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170289    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170217    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170630    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170711    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.17067658 +0000 UTC m=+5.020189546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170716    5028 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170759    5028 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170797    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert podName:69e320d2-2d3d-45fb-a44b-41bfe19091ae nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.170780992 +0000 UTC m=+5.020293944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-g99mq" (UID: "69e320d2-2d3d-45fb-a44b-41bfe19091ae") : secret "networking-console-plugin-cert" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170320    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170817    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.170807677 +0000 UTC m=+5.020320628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170719    5028 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170836    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170843    5028 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170854    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls podName:a461f5f1-641c-4f5e-8f29-995036f2f0f7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.170844835 +0000 UTC m=+5.020357817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qhkhv" (UID: "a461f5f1-641c-4f5e-8f29-995036f2f0f7") : secret "samples-operator-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.170849    5028 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170877    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.170864184 +0000 UTC m=+5.020377167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170602    5028 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170890    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.170883796 +0000 UTC m=+5.020396746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170917    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.170904452 +0000 UTC m=+5.020417426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170923    5028 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:01:12.171047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:12.170954    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs podName:d91ca99d-6276-4875-9d6b-e1e49533a930 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:14.17094454 +0000 UTC m=+5.020457521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs") pod "network-metrics-daemon-hfs8l" (UID: "d91ca99d-6276-4875-9d6b-e1e49533a930") : secret "metrics-daemon-secret" not found
Oct 30 22:01:12.173550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.173526    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nxvq5\" (UID: \"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9\") " pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:12.193238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.193210    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nxvq5"
Oct 30 22:01:12.372025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.371999    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:01:12.375357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.375331    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.522498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.522471    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gpmjd_e7f71a60-c64f-4960-934a-6c75e8d94546/node-ca/0.log"
Oct 30 22:01:12.579344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.579300    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.626705 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.625958    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nxvq5"]
Oct 30 22:01:12.637424 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:12.637393    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a05cb9e_8878_49ff_8a1a_b780fb4c6cd9.slice/crio-280c29ea42e6f15baf32f4e85dc125a4d707722b6e921042b105e6f8303988e1 WatchSource:0}: Error finding container 280c29ea42e6f15baf32f4e85dc125a4d707722b6e921042b105e6f8303988e1: Status 404 returned error can't find the container with id 280c29ea42e6f15baf32f4e85dc125a4d707722b6e921042b105e6f8303988e1
Oct 30 22:01:12.944100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:12.944007    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:13.212507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.212471    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:13.323053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.323021    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s96ng_914a456e-5637-4208-98bd-ee4078634291/serve-healthcheck-canary/0.log"
Oct 30 22:01:13.375397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.375368    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:13.377762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.377722    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxvq5" event={"ID":"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9","Type":"ContainerStarted","Data":"50d76edb2da2a65430851d9a1854d70100ed8a93568310ae0adcf2db527dd8e4"}
Oct 30 22:01:13.378210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.378154    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxvq5" event={"ID":"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9","Type":"ContainerStarted","Data":"280c29ea42e6f15baf32f4e85dc125a4d707722b6e921042b105e6f8303988e1"}
Oct 30 22:01:13.383379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.383346    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-x6s86" event={"ID":"bd3282fe-9d41-4c82-9bac-bb74c4209b62","Type":"ContainerStarted","Data":"e95be75db14d7e7d5603d10aba2dd1ef2d6784c1df684cf3eac10a8e8a247c0c"}
Oct 30 22:01:13.384722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.384682    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n8qq6" event={"ID":"43829ec3-a430-4d2e-b2ee-19ae8df72ac9","Type":"ContainerStarted","Data":"0932b2c3c715d628eed6e406e4485949d80ef850580599c8a1ff026ef0a2de1d"}
Oct 30 22:01:13.524438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.524363    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-n8qq6_43829ec3-a430-4d2e-b2ee-19ae8df72ac9/migrator/0.log"
Oct 30 22:01:13.548629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.548599    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:13.721781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.721746    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-x6s86_bd3282fe-9d41-4c82-9bac-bb74c4209b62/migrator/0.log"
Oct 30 22:01:13.923508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:13.923477    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-d6swh_f572fd28-f04a-45ed-a153-5921471fb276/kube-storage-version-migrator-operator/0.log"
Oct 30 22:01:14.190060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.189973    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:14.190060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190020    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:14.190060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190049    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190068    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190089    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190114    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190163    5028 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190223    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190242    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190170    5028 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190249    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190296    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:18.190277076 +0000 UTC m=+9.039790041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190328    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:18.190317582 +0000 UTC m=+9.039830537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190331    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190341    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:14.190357 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190346    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls podName:f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf nodeName:}" failed. No retries permitted until 2026-10-30 22:01:18.190337896 +0000 UTC m=+9.039850850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls") pod "dns-default-zzj6z" (UID: "f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf") : secret "dns-default-metrics-tls" not found
Oct 30 22:01:14.190852 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190390    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:18.190377857 +0000 UTC m=+9.039890813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:14.190852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.190386    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:14.190852 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190527    5028 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:14.190852 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:14.190575    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls podName:73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:18.190562197 +0000 UTC m=+9.040075149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2hrpg" (UID: "73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:01:14.192601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.192576    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69e320d2-2d3d-45fb-a44b-41bfe19091ae-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-g99mq\" (UID: \"69e320d2-2d3d-45fb-a44b-41bfe19091ae\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:14.192709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.192604    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91ca99d-6276-4875-9d6b-e1e49533a930-metrics-certs\") pod \"network-metrics-daemon-hfs8l\" (UID: \"d91ca99d-6276-4875-9d6b-e1e49533a930\") " pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:14.192748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.192711    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a461f5f1-641c-4f5e-8f29-995036f2f0f7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qhkhv\" (UID: \"a461f5f1-641c-4f5e-8f29-995036f2f0f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:14.281803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.281771    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"
Oct 30 22:01:14.282856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.282835    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq"
Oct 30 22:01:14.288974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.288956    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hfs8l"
Oct 30 22:01:14.352314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.344553    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:14.390017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.389988    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxvq5" event={"ID":"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9","Type":"ContainerStarted","Data":"b593cbaedcc48994b8ad346784986402eae538e8aeec028b71cf4737e10e6301"}
Oct 30 22:01:14.660156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.660116    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-g99mq"]
Oct 30 22:01:14.695804 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:14.695768    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e320d2_2d3d_45fb_a44b_41bfe19091ae.slice/crio-97104034007c76c4833aa31b9acb4015a4d52c2151941455b18e4adea856c4db WatchSource:0}: Error finding container 97104034007c76c4833aa31b9acb4015a4d52c2151941455b18e4adea856c4db: Status 404 returned error can't find the container with id 97104034007c76c4833aa31b9acb4015a4d52c2151941455b18e4adea856c4db
Oct 30 22:01:14.705731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.705678    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hfs8l"]
Oct 30 22:01:14.708905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.708879    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv"]
Oct 30 22:01:14.746910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.746886    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:14.941076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:14.940989    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:15.139323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.139284    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:15.396594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.396554    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq" event={"ID":"69e320d2-2d3d-45fb-a44b-41bfe19091ae","Type":"ContainerStarted","Data":"97104034007c76c4833aa31b9acb4015a4d52c2151941455b18e4adea856c4db"}
Oct 30 22:01:15.398113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.398079    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv" event={"ID":"a461f5f1-641c-4f5e-8f29-995036f2f0f7","Type":"ContainerStarted","Data":"c339dc5ec4167d78737cfd68cd6ba1fd259177f6fd55590a564b0a4a4ba34d07"}
Oct 30 22:01:15.399318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.399290    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hfs8l" event={"ID":"d91ca99d-6276-4875-9d6b-e1e49533a930","Type":"ContainerStarted","Data":"a06ba9fba6764aab66c731e709b27d758e00b3008c6e7f400de692dd69381c5b"}
Oct 30 22:01:15.551774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.551737    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:15.601778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.601736    5028 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:15.602191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:15.602173    5028 scope.go:122] "RemoveContainer" containerID="773bce85c4f07b008782382cd84840b2ad511c01786eca90cebcb89c37344991"
Oct 30 22:01:15.602396 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:15.602375    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q2lps_openshift-console-operator(e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" podUID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b"
Oct 30 22:01:16.028484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:16.028446    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4j6gf"
Oct 30 22:01:16.353793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:16.353680    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:16.551471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:16.551438    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:16.993844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:16.993797    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:17.170437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:17.170345    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:17.411030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:17.410993    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-g99mq" event={"ID":"69e320d2-2d3d-45fb-a44b-41bfe19091ae","Type":"ContainerStarted","Data":"92a09a1343db2142a9bed29e94404e770c0bc3306980be667c8759886d7cd306"}
Oct 30 22:01:17.413015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:17.412985    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nxvq5" event={"ID":"2a05cb9e-8878-49ff-8a1a-b780fb4c6cd9","Type":"ContainerStarted","Data":"a5051e70a1e61b564d601cbfdb38b472f33d2f7ddd427b73b0e688b1a862c570"}
Oct 30 22:01:17.414737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:17.414715    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hfs8l" event={"ID":"d91ca99d-6276-4875-9d6b-e1e49533a930","Type":"ContainerStarted","Data":"54d0fe3aa7eb41f39134df18d630ff29e3f6130b239cc8f4f7f21a4bcd9b81d1"}
Oct 30 22:01:17.414832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:17.414758    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hfs8l" event={"ID":"d91ca99d-6276-4875-9d6b-e1e49533a930","Type":"ContainerStarted","Data":"b4e8fb166e9ca29808ad67561ef46f1f97ae291416560a23f4a82c018ed5cace"}
Oct 30 22:01:17.757799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:17.757766    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:18.229192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.229153    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:18.229408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.229205    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:18.229408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.229225    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:18.229408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.229256    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:18.229408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.229296    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:18.229408 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229337    5028 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229412    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs podName:56a790e5-683e-4dfd-82e1-1bde4629e09f nodeName:}" failed. No retries permitted until 2026-10-30 22:01:26.229391186 +0000 UTC m=+17.078904149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs") pod "router-default-86b69957bf-6kffb" (UID: "56a790e5-683e-4dfd-82e1-1bde4629e09f") : secret "router-metrics-certs-default" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229413    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229434    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-876dbf8dd-czb2r: secret "image-registry-tls" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229486    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls podName:7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:26.229474803 +0000 UTC m=+17.078987757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls") pod "image-registry-876dbf8dd-czb2r" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59") : secret "image-registry-tls" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229343    5028 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229512    5028 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d45ff7cc7-twst9: secret "image-registry-tls" not found
Oct 30 22:01:18.229665 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.229567    5028 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls podName:051a42ce-76bb-4e7e-a32c-82bff50cd38d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:26.229550721 +0000 UTC m=+17.079063672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls") pod "image-registry-6d45ff7cc7-twst9" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d") : secret "image-registry-tls" not found
Oct 30 22:01:18.231618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.231599    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf-metrics-tls\") pod \"dns-default-zzj6z\" (UID: \"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf\") " pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:18.231728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.231644    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2hrpg\" (UID: \"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:18.373406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.373368    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:18.419739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.419707    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv" event={"ID":"a461f5f1-641c-4f5e-8f29-995036f2f0f7","Type":"ContainerStarted","Data":"dd7781d46acac297998977eea7e1d058878a79e0cb663804602fd79223b45d6a"}
Oct 30 22:01:18.419888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.419755    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qhkhv" event={"ID":"a461f5f1-641c-4f5e-8f29-995036f2f0f7","Type":"ContainerStarted","Data":"877607be0d6a03e6e480659a6653d4a661f99b84b8c81fedfc5fa5ca4ca3cad5"}
Oct 30 22:01:18.478354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.478318    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"
Oct 30 22:01:18.491794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.491751    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:18.558951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.558805    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:18.635702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.635646    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:18.636085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.636063    5028 scope.go:122] "RemoveContainer" containerID="773bce85c4f07b008782382cd84840b2ad511c01786eca90cebcb89c37344991"
Oct 30 22:01:18.636304 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:18.636284    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q2lps_openshift-console-operator(e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" podUID="e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b"
Oct 30 22:01:18.708214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.708152    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:18.874210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.874185    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg"]
Oct 30 22:01:18.876634 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:18.876608    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ea9998_7cb8_45d2_ad2b_473bd8f4ae7e.slice/crio-21eab7ae427c685d21cf2d48dfd381d8f51b493beb03b5d67a65ba4df616be33 WatchSource:0}: Error finding container 21eab7ae427c685d21cf2d48dfd381d8f51b493beb03b5d67a65ba4df616be33: Status 404 returned error can't find the container with id 21eab7ae427c685d21cf2d48dfd381d8f51b493beb03b5d67a65ba4df616be33
Oct 30 22:01:18.895415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.895391    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zzj6z"]
Oct 30 22:01:18.897012 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:18.896977    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e6ad1e_6b40_40b5_bb22_8d13dcdaa7bf.slice/crio-f494c629ea698cc01e50a239b5cacf42054313fe115ba041f67943136272d31e WatchSource:0}: Error finding container f494c629ea698cc01e50a239b5cacf42054313fe115ba041f67943136272d31e: Status 404 returned error can't find the container with id f494c629ea698cc01e50a239b5cacf42054313fe115ba041f67943136272d31e
Oct 30 22:01:18.953912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:18.953882    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:19.146457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:19.146373    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:19.425889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:19.425800    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg" event={"ID":"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e","Type":"ContainerStarted","Data":"21eab7ae427c685d21cf2d48dfd381d8f51b493beb03b5d67a65ba4df616be33"}
Oct 30 22:01:19.427028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:19.426995    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzj6z" event={"ID":"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf","Type":"ContainerStarted","Data":"f494c629ea698cc01e50a239b5cacf42054313fe115ba041f67943136272d31e"}
Oct 30 22:01:19.741480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:19.741448    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.192042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.191966    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:20.348336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.348296    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.533190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.533156    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:20.533408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.533392    5028 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:01:20.553083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.553050    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.698571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.698533    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.871914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:20.871836    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:21.020736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.020702    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:21.180735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.180704    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:21.325381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.325336    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:21.438912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.438883    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzj6z" event={"ID":"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf","Type":"ContainerStarted","Data":"76f1cec59557d8482d8625e6a5a9f2c1f98c5961a985975859c449d9ce2c7061"}
Oct 30 22:01:21.439012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.438928    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zzj6z" event={"ID":"f1e6ad1e-6b40-40b5-bb22-8d13dcdaa7bf","Type":"ContainerStarted","Data":"f8e0936ea5485286c0faadba475e449df471fb203a4101d76c8a5ebfeba44510"}
Oct 30 22:01:21.439012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.438967    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:21.480477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:21.480440    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:22.059797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.059739    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:22.284179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.284144    5028 scope.go:122] "RemoveContainer" containerID="773bce85c4f07b008782382cd84840b2ad511c01786eca90cebcb89c37344991"
Oct 30 22:01:22.443771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.443733    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2hrpg" event={"ID":"73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e","Type":"ContainerStarted","Data":"38814a38aea0ae0dece949ec1c5cb9e1de287a9fb0192b921401178f8001913e"}
Oct 30 22:01:22.445650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.445627    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:01:22.445797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.445748    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps" event={"ID":"e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b","Type":"ContainerStarted","Data":"d3ebe556297ee4f23b4b40fa20bfa79769bfd4dc6c6c91c28a8c2b54175c1f4f"}
Oct 30 22:01:22.494003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.493969    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:22.643608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.643522    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:22.787780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:22.787745    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:23.504868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:23.504834    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:23.677771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:23.677734    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:23.847718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:23.847611    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:23.994388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:23.994355    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:24.166047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:24.165975    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:24.310896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:24.310861    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:25.086226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:25.086191    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.261102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.261069    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.303980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.303943    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:26.304141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.303993    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:26.304141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.304117    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:26.306501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.306469    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a790e5-683e-4dfd-82e1-1bde4629e09f-metrics-certs\") pod \"router-default-86b69957bf-6kffb\" (UID: \"56a790e5-683e-4dfd-82e1-1bde4629e09f\") " pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:26.306615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.306501    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"image-registry-6d45ff7cc7-twst9\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") " pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:26.306615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.306473    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"image-registry-876dbf8dd-czb2r\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") " pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:26.405394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.405357    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.578174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.578094    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:26.581286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.581262    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:26.591482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.591453    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:26.684166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.684103    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d45ff7cc7-twst9"]
Oct 30 22:01:26.718150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.718117    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.866547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:26.866473    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:27.017206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.017165    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:27.028979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.028817    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-876dbf8dd-czb2r"]
Oct 30 22:01:27.031575 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:27.031540    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f37686f_eba2_4b8a_bd8f_ca6c2fe38a59.slice/crio-f0ec4654364f220b51a9d803e4c87997706b360162f13a8631f4b8dcb02493db WatchSource:0}: Error finding container f0ec4654364f220b51a9d803e4c87997706b360162f13a8631f4b8dcb02493db: Status 404 returned error can't find the container with id f0ec4654364f220b51a9d803e4c87997706b360162f13a8631f4b8dcb02493db
Oct 30 22:01:27.074536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.074414    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d45ff7cc7-twst9"]
Oct 30 22:01:27.077169 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:27.077137    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051a42ce_76bb_4e7e_a32c_82bff50cd38d.slice/crio-2f6365e5a7e34d55b8cf60c23111cfa9af945e6b1c8cca902812d689ec785021 WatchSource:0}: Error finding container 2f6365e5a7e34d55b8cf60c23111cfa9af945e6b1c8cca902812d689ec785021: Status 404 returned error can't find the container with id 2f6365e5a7e34d55b8cf60c23111cfa9af945e6b1c8cca902812d689ec785021
Oct 30 22:01:27.077415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.077295    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86b69957bf-6kffb"]
Oct 30 22:01:27.079559 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:27.079535    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a790e5_683e_4dfd_82e1_1bde4629e09f.slice/crio-db2a8fa0245818a92590d187271278879cd7a3c97584452e94ef8d6a8230603a WatchSource:0}: Error finding container db2a8fa0245818a92590d187271278879cd7a3c97584452e94ef8d6a8230603a: Status 404 returned error can't find the container with id db2a8fa0245818a92590d187271278879cd7a3c97584452e94ef8d6a8230603a
Oct 30 22:01:27.182137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.182097    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:27.336791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.336754    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:27.458928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.458899    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tw7xp"
Oct 30 22:01:27.468157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.468117    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9" event={"ID":"051a42ce-76bb-4e7e-a32c-82bff50cd38d","Type":"ContainerStarted","Data":"48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a"}
Oct 30 22:01:27.468317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.468165    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9" event={"ID":"051a42ce-76bb-4e7e-a32c-82bff50cd38d","Type":"ContainerStarted","Data":"2f6365e5a7e34d55b8cf60c23111cfa9af945e6b1c8cca902812d689ec785021"}
Oct 30 22:01:27.468317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.468258    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:27.469755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.469728    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r" event={"ID":"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59","Type":"ContainerStarted","Data":"fd01a4c9dc5f5b9602d0267072dd1dcf03aa1446d30955ed6746d3e4db2fbee3"}
Oct 30 22:01:27.469877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.469769    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r" event={"ID":"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59","Type":"ContainerStarted","Data":"f0ec4654364f220b51a9d803e4c87997706b360162f13a8631f4b8dcb02493db"}
Oct 30 22:01:27.469877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.469799    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:27.471328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.471302    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86b69957bf-6kffb" event={"ID":"56a790e5-683e-4dfd-82e1-1bde4629e09f","Type":"ContainerStarted","Data":"654a2fd933a511ea973d7fe9d8cc36a651b0854e33b23969d2c496f782fa67f6"}
Oct 30 22:01:27.471431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.471343    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86b69957bf-6kffb" event={"ID":"56a790e5-683e-4dfd-82e1-1bde4629e09f","Type":"ContainerStarted","Data":"db2a8fa0245818a92590d187271278879cd7a3c97584452e94ef8d6a8230603a"}
Oct 30 22:01:27.488244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.488221    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:27.578775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.578740    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:27.580980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.580948    5028 patch_prober.go:29] interesting pod/router-default-86b69957bf-6kffb container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 30 22:01:27.580980 ip-10-0-129-125 kubenswrapper[5028]: [+]has-synced ok
Oct 30 22:01:27.580980 ip-10-0-129-125 kubenswrapper[5028]: [+]process-running ok
Oct 30 22:01:27.580980 ip-10-0-129-125 kubenswrapper[5028]: healthz check failed
Oct 30 22:01:27.581246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:27.581008    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-86b69957bf-6kffb" podUID="56a790e5-683e-4dfd-82e1-1bde4629e09f" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 30 22:01:28.126094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:28.126060    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:28.273888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:28.273834    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:28.434780 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:28.434701    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:01:28.532813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:28.532769    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:28.579997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:28.579965    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:28.681190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:28.681155    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:29.480998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:29.480960    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:29.482196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:29.482171    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86b69957bf-6kffb"
Oct 30 22:01:31.448202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:31.448171    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zzj6z"
Oct 30 22:01:32.447061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:32.447016    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:32.451137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:32.451110    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q2lps"
Oct 30 22:01:33.532931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:33.532895    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:33.681608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:33.681571    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:35.776912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:35.776878    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:35.938504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:35.938474    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:36.087889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:36.087805    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:36.244108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:36.244070    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.419647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:41.419610    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.657784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:41.657739    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.810081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:41.810048    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.128981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.128887    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.305845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.305809    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.455389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.455289    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.605093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.605062    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.636195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.636163    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tcd2z"]
Oct 30 22:01:42.695518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.695486    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.751371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.751340    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.766142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.766113    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 22:01:42.792918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.792887    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qp8bv\""
Oct 30 22:01:42.793089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.792949    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 22:01:42.803210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.803182    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 22:01:42.803308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.803214    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 22:01:42.844753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844718    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsjm\" (UniqueName: \"kubernetes.io/projected/10a9c7f4-272c-472a-acd1-b76ff3e29846-kube-api-access-jmsjm\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.844753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844755    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-accelerators-collector-config\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.844973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844785    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-root\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.844973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844907    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9c7f4-272c-472a-acd1-b76ff3e29846-metrics-client-ca\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.844973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844940    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-sys\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.844973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844960    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-textfile\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.845129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.844976    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-tls\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.845129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.845003    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-wtmp\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.845129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.845080    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.915799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.915767    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:42.946225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946189    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsjm\" (UniqueName: \"kubernetes.io/projected/10a9c7f4-272c-472a-acd1-b76ff3e29846-kube-api-access-jmsjm\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946232    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-accelerators-collector-config\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946275    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-root\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946322    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9c7f4-272c-472a-acd1-b76ff3e29846-metrics-client-ca\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946354    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-sys\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946382    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-textfile\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946408    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-tls\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946429    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-root\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946508    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-sys\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946449    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-wtmp\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946589    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.946967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.946949    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-textfile\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.947102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.947084    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-wtmp\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.947178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.947094    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10a9c7f4-272c-472a-acd1-b76ff3e29846-metrics-client-ca\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.947543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.947520    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-accelerators-collector-config\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.949797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.949774    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:42.951284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:42.951265    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10a9c7f4-272c-472a-acd1-b76ff3e29846-node-exporter-tls\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:43.005893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:43.005817    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsjm\" (UniqueName: \"kubernetes.io/projected/10a9c7f4-272c-472a-acd1-b76ff3e29846-kube-api-access-jmsjm\") pod \"node-exporter-tcd2z\" (UID: \"10a9c7f4-272c-472a-acd1-b76ff3e29846\") " pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:43.043760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:43.043726    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tcd2z"
Oct 30 22:01:43.059863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:43.059826    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:43.073023 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:01:43.072988    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a9c7f4_272c_472a_acd1_b76ff3e29846.slice/crio-11ce7f4b535eb3e30f0bba08b8fe4a88d8a58ed8adf6ffe509e6a06825b18417 WatchSource:0}: Error finding container 11ce7f4b535eb3e30f0bba08b8fe4a88d8a58ed8adf6ffe509e6a06825b18417: Status 404 returned error can't find the container with id 11ce7f4b535eb3e30f0bba08b8fe4a88d8a58ed8adf6ffe509e6a06825b18417
Oct 30 22:01:43.217498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:43.217464    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:43.378471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:43.378398    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lm2gp"
Oct 30 22:01:43.537874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:43.537837    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tcd2z" event={"ID":"10a9c7f4-272c-472a-acd1-b76ff3e29846","Type":"ContainerStarted","Data":"11ce7f4b535eb3e30f0bba08b8fe4a88d8a58ed8adf6ffe509e6a06825b18417"}
Oct 30 22:01:44.133948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:44.133907    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:44.288476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:44.288438    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:44.452479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:44.452355    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:44.967248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:44.967212    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:45.114468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:45.114379    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:45.547814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:45.547777    5028 generic.go:356] "Generic (PLEG): container finished" podID="10a9c7f4-272c-472a-acd1-b76ff3e29846" containerID="379bba0039b2d88f4093b8220d6f6c8a3d464970469cb7c707f760a8799e639d" exitCode=0
Oct 30 22:01:45.548003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:45.547868    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tcd2z" event={"ID":"10a9c7f4-272c-472a-acd1-b76ff3e29846","Type":"ContainerDied","Data":"379bba0039b2d88f4093b8220d6f6c8a3d464970469cb7c707f760a8799e639d"}
Oct 30 22:01:45.791023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:45.790987    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:45.942303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:45.942225    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:46.554208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:46.554171    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tcd2z" event={"ID":"10a9c7f4-272c-472a-acd1-b76ff3e29846","Type":"ContainerStarted","Data":"f4324ab1dd61611d90c27c0e72238252854a0dc5c13bd05857fc70e214fde740"}
Oct 30 22:01:46.554606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:46.554225    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tcd2z" event={"ID":"10a9c7f4-272c-472a-acd1-b76ff3e29846","Type":"ContainerStarted","Data":"42281efa2c1814efa1f196f432b02dc1f80e1ddcc8c30294139e963e1cb8d5a2"}
Oct 30 22:01:46.759733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:46.759663    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tcd2z" podStartSLOduration=3.174974296 podStartE2EDuration="4.759648971s" totalImagesPullingTime="1.515590194s" totalInitContainerRuntime="69.084481ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:46.716246012 +0000 UTC m=+37.565758986" watchObservedRunningTime="2026-10-30 22:01:46.759648971 +0000 UTC m=+37.609161945"
Oct 30 22:01:46.780962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:46.780926    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:46.998350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:46.998306    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:47.171462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:47.171428    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:47.329224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:47.329146    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:47.474428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:47.474397    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:47.510786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:47.510749    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:48.483156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:48.483126    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:01:48.550891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:48.550855    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:48.699780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:48.699745    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.195903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:49.195806    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.345143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:49.345105    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.761554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:49.761522    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.998443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:49.998414    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:51.618266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:51.618231    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:52.487102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.487054    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9" podUID="051a42ce-76bb-4e7e-a32c-82bff50cd38d" containerName="registry" containerID="cri-o://48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a" gracePeriod=30
Oct 30 22:01:52.735641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.735616    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:52.742328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742264    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742321    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742342    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742458    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742520    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742554    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742584    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742617    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") pod \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\" (UID: \"051a42ce-76bb-4e7e-a32c-82bff50cd38d\") "
Oct 30 22:01:52.742957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.742901    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:52.743080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.743005    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:52.744942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.744915    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:52.745194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.745160    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:52.745299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.745184    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "kube-api-access-7nkvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:52.745299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.745278    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:52.745409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.745359    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:52.749871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.749760    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted" pod "051a42ce-76bb-4e7e-a32c-82bff50cd38d" (UID: "051a42ce-76bb-4e7e-a32c-82bff50cd38d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:52.844188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844155    5028 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-bound-sa-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844184    5028 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/051a42ce-76bb-4e7e-a32c-82bff50cd38d-ca-trust-extracted\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844194    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nkvm\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-kube-api-access-7nkvm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844206    5028 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-trusted-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844215    5028 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-installation-pull-secrets\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844224    5028 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-certificates\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844232    5028 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/051a42ce-76bb-4e7e-a32c-82bff50cd38d-registry-tls\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.844484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:52.844241    5028 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/051a42ce-76bb-4e7e-a32c-82bff50cd38d-image-registry-private-configuration\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:53.587230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.587146    5028 generic.go:356] "Generic (PLEG): container finished" podID="051a42ce-76bb-4e7e-a32c-82bff50cd38d" containerID="48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a" exitCode=0
Oct 30 22:01:53.587230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.587190    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9"
Oct 30 22:01:53.587409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.587196    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9" event={"ID":"051a42ce-76bb-4e7e-a32c-82bff50cd38d","Type":"ContainerDied","Data":"48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a"}
Oct 30 22:01:53.587409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.587305    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d45ff7cc7-twst9" event={"ID":"051a42ce-76bb-4e7e-a32c-82bff50cd38d","Type":"ContainerDied","Data":"2f6365e5a7e34d55b8cf60c23111cfa9af945e6b1c8cca902812d689ec785021"}
Oct 30 22:01:53.587409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.587334    5028 scope.go:122] "RemoveContainer" containerID="48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a"
Oct 30 22:01:53.595774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.595749    5028 scope.go:122] "RemoveContainer" containerID="48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a"
Oct 30 22:01:53.596088 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:53.596068    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a\": container with ID starting with 48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a not found: ID does not exist" containerID="48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a"
Oct 30 22:01:53.596170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.596097    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a"} err="failed to get container status \"48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a\": rpc error: code = NotFound desc = could not find container \"48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a\": container with ID starting with 48608f1e1c1e22a6de3296668f2ace41f404c64453a0199e2c063894ba06d80a not found: ID does not exist"
Oct 30 22:01:53.798306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.798271    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d45ff7cc7-twst9"]
Oct 30 22:01:53.845615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:53.845537    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d45ff7cc7-twst9"]
Oct 30 22:01:54.873578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:54.873545    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:55.046894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:55.046861    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:55.288765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:55.288721    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051a42ce-76bb-4e7e-a32c-82bff50cd38d" path="/var/lib/kubelet/pods/051a42ce-76bb-4e7e-a32c-82bff50cd38d/volumes"
Oct 30 22:01:58.435014 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:01:58.434969    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:01:58.571346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:58.571313    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-876dbf8dd-czb2r"]
Oct 30 22:01:58.715603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:58.715570    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:58.876927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:58.876893    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:59.270073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:59.270039    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:59.449953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:01:59.449922    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:00.651836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:00.651800    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:00.800875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:00.800838    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:01.264273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:01.264230    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:01.414145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:01.414106    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:02.473191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:02.473148    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:02.619529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:02.619491    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:03.603106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:03.603073    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:03.752864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:03.752832    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:09.092017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:09.091971    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:09.246024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:09.245991    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:18.601924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:18.601880    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:18.749905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:18.749876    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:19.700980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:19.700942    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:19.848596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:19.848565    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:21.694109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:21.694074    5028 generic.go:356] "Generic (PLEG): container finished" podID="f572fd28-f04a-45ed-a153-5921471fb276" containerID="2d8f137ac1aa97f0889730ddccdb7ad0db0759aafa05ce76370dbd569a32b87a" exitCode=0
Oct 30 22:02:21.694533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:21.694152    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh" event={"ID":"f572fd28-f04a-45ed-a153-5921471fb276","Type":"ContainerDied","Data":"2d8f137ac1aa97f0889730ddccdb7ad0db0759aafa05ce76370dbd569a32b87a"}
Oct 30 22:02:21.694533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:21.694496    5028 scope.go:122] "RemoveContainer" containerID="2d8f137ac1aa97f0889730ddccdb7ad0db0759aafa05ce76370dbd569a32b87a"
Oct 30 22:02:22.699921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:22.699887    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d6swh" event={"ID":"f572fd28-f04a-45ed-a153-5921471fb276","Type":"ContainerStarted","Data":"e02cf4b385f4299d7aee5c90c90e99c508d90bda422d6cacf9ee1451672cfc35"}
Oct 30 22:02:23.591105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.591059    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r" podUID="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" containerName="registry" containerID="cri-o://fd01a4c9dc5f5b9602d0267072dd1dcf03aa1446d30955ed6746d3e4db2fbee3" gracePeriod=30
Oct 30 22:02:23.706460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.706427    5028 generic.go:356] "Generic (PLEG): container finished" podID="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" containerID="fd01a4c9dc5f5b9602d0267072dd1dcf03aa1446d30955ed6746d3e4db2fbee3" exitCode=0
Oct 30 22:02:23.707035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.706496    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r" event={"ID":"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59","Type":"ContainerDied","Data":"fd01a4c9dc5f5b9602d0267072dd1dcf03aa1446d30955ed6746d3e4db2fbee3"}
Oct 30 22:02:23.837038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.837012    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:02:23.905473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905394    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905433    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905470    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905491    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905516    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905564    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905587    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.905760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905617    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") pod \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\" (UID: \"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59\") "
Oct 30 22:02:23.906091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.905948    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:23.906442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.906408    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:23.908012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.907984    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:23.908177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.908135    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:23.908448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.908419    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "kube-api-access-cnnjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:23.908528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.908460    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:23.908528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.908486    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:23.913394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:23.913371    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted" pod "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" (UID: "7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:24.006253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006218    5028 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-certificates\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006245    5028 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-ca-trust-extracted\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006255    5028 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-installation-pull-secrets\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006265    5028 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-trusted-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006274    5028 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-bound-sa-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006283    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnjx\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-kube-api-access-cnnjx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006292    5028 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-registry-tls\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.006498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.006301    5028 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59-image-registry-private-configuration\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.712137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.712098    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r"
Oct 30 22:02:24.712579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.712098    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-876dbf8dd-czb2r" event={"ID":"7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59","Type":"ContainerDied","Data":"f0ec4654364f220b51a9d803e4c87997706b360162f13a8631f4b8dcb02493db"}
Oct 30 22:02:24.712579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.712238    5028 scope.go:122] "RemoveContainer" containerID="fd01a4c9dc5f5b9602d0267072dd1dcf03aa1446d30955ed6746d3e4db2fbee3"
Oct 30 22:02:24.916996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.916955    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-876dbf8dd-czb2r"]
Oct 30 22:02:24.924656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:24.924627    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-876dbf8dd-czb2r"]
Oct 30 22:02:25.291403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:25.291359    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" path="/var/lib/kubelet/pods/7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59/volumes"
Oct 30 22:02:28.433466 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:02:28.433423    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:02:30.713982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:30.713946    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:30.740797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:30.740763    5028 generic.go:356] "Generic (PLEG): container finished" podID="be68f52a-8bf4-474e-b28d-133aecbfb229" containerID="62295b7387f480a1535263f591ed4a87bdfc7e5d42f3bba5abe18a425ea31049" exitCode=0
Oct 30 22:02:30.740961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:30.740839    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cxs8w" event={"ID":"be68f52a-8bf4-474e-b28d-133aecbfb229","Type":"ContainerDied","Data":"62295b7387f480a1535263f591ed4a87bdfc7e5d42f3bba5abe18a425ea31049"}
Oct 30 22:02:30.741220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:30.741205    5028 scope.go:122] "RemoveContainer" containerID="62295b7387f480a1535263f591ed4a87bdfc7e5d42f3bba5abe18a425ea31049"
Oct 30 22:02:31.746076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:31.746040    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-cxs8w" event={"ID":"be68f52a-8bf4-474e-b28d-133aecbfb229","Type":"ContainerStarted","Data":"deb2ece810dcb90e2a96639cd93ef13bbba816b2d1ca99801ed475a71b7e0df9"}
Oct 30 22:02:31.908299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:31.908265    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-slw2d-b5qs2"]
Oct 30 22:02:31.908514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:31.908491    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-slw2d-b5qs2" podUID="4b72179a-878b-4792-939e-6db6499f00bd" containerName="insights-gathering" containerID="cri-o://edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f" gracePeriod=30
Oct 30 22:02:32.157241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.157216    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:02:32.274674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.274592    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") pod \"4b72179a-878b-4792-939e-6db6499f00bd\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") "
Oct 30 22:02:32.274846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.274716    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") pod \"4b72179a-878b-4792-939e-6db6499f00bd\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") "
Oct 30 22:02:32.274846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.274765    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") pod \"4b72179a-878b-4792-939e-6db6499f00bd\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") "
Oct 30 22:02:32.274846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.274820    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") pod \"4b72179a-878b-4792-939e-6db6499f00bd\" (UID: \"4b72179a-878b-4792-939e-6db6499f00bd\") "
Oct 30 22:02:32.275023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.274989    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle" pod "4b72179a-878b-4792-939e-6db6499f00bd" (UID: "4b72179a-878b-4792-939e-6db6499f00bd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:32.275155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.275083    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path" pod "4b72179a-878b-4792-939e-6db6499f00bd" (UID: "4b72179a-878b-4792-939e-6db6499f00bd"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:32.275208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.275182    5028 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-trusted-ca-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.275257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.275241    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle" pod "4b72179a-878b-4792-939e-6db6499f00bd" (UID: "4b72179a-878b-4792-939e-6db6499f00bd"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:32.277187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.277157    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt" pod "4b72179a-878b-4792-939e-6db6499f00bd" (UID: "4b72179a-878b-4792-939e-6db6499f00bd"). InnerVolumeSpecName "kube-api-access-p2fzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:32.376040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.375994    5028 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b72179a-878b-4792-939e-6db6499f00bd-archives-path\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.376040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.376035    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fzt\" (UniqueName: \"kubernetes.io/projected/4b72179a-878b-4792-939e-6db6499f00bd-kube-api-access-p2fzt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.376040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.376046    5028 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b72179a-878b-4792-939e-6db6499f00bd-service-ca-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.751507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.751475    5028 generic.go:356] "Generic (PLEG): container finished" podID="4b72179a-878b-4792-939e-6db6499f00bd" containerID="edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f" exitCode=2
Oct 30 22:02:32.751919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.751523    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slw2d-b5qs2"
Oct 30 22:02:32.751919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.751536    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slw2d-b5qs2" event={"ID":"4b72179a-878b-4792-939e-6db6499f00bd","Type":"ContainerDied","Data":"edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f"}
Oct 30 22:02:32.751919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.751579    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slw2d-b5qs2" event={"ID":"4b72179a-878b-4792-939e-6db6499f00bd","Type":"ContainerDied","Data":"b22e2c211049fc4364d31170366e2204480a3415b28859cce040599c49e655cf"}
Oct 30 22:02:32.751919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.751598    5028 scope.go:122] "RemoveContainer" containerID="edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f"
Oct 30 22:02:32.759772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.759752    5028 scope.go:122] "RemoveContainer" containerID="edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f"
Oct 30 22:02:32.760068 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:02:32.760051    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f\": container with ID starting with edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f not found: ID does not exist" containerID="edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f"
Oct 30 22:02:32.760116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.760081    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f"} err="failed to get container status \"edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f\": rpc error: code = NotFound desc = could not find container \"edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f\": container with ID starting with edc37578358351b1e69a1f676985b34e8e44311ab9839e14c6964a1c43f5198f not found: ID does not exist"
Oct 30 22:02:32.907518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.907487    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-slw2d-b5qs2"]
Oct 30 22:02:32.954757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:32.954722    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-slw2d-b5qs2"]
Oct 30 22:02:33.288366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:33.288335    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b72179a-878b-4792-939e-6db6499f00bd" path="/var/lib/kubelet/pods/4b72179a-878b-4792-939e-6db6499f00bd/volumes"
Oct 30 22:02:36.768593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:36.768555    5028 generic.go:356] "Generic (PLEG): container finished" podID="b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4" containerID="e67178adae82286e7154f1d66f4528428b34e82018b335e7e92f0d073c675174" exitCode=0
Oct 30 22:02:36.768996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:36.768627    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w" event={"ID":"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4","Type":"ContainerDied","Data":"e67178adae82286e7154f1d66f4528428b34e82018b335e7e92f0d073c675174"}
Oct 30 22:02:36.768996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:36.768979    5028 scope.go:122] "RemoveContainer" containerID="e67178adae82286e7154f1d66f4528428b34e82018b335e7e92f0d073c675174"
Oct 30 22:02:37.774327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:37.774291    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-m592w" event={"ID":"b8e6d4d4-dc9b-44be-8dc1-abc723e7ffe4","Type":"ContainerStarted","Data":"3c9439f4c7702457f8786f470cd1d99ea5dee84e00972a3dbd1a1d8624f2c7f8"}
Oct 30 22:02:51.391369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:51.391327    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:52.606838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:52.606805    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:52.753741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:52.753700    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:53.021512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:53.021480    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:53.186862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:53.186816    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:58.434545 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:02:58.434507    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:02:59.709948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:02:59.709917    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:04.008006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:03:04.007973    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:04.164193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:03:04.164151    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:18.636527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:03:18.636483    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:18.786404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:03:18.786373    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:20.323681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:03:20.323583    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:28.433878 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:03:28.433840    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:03:40.965301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:03:40.965262    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:58.434417 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:03:58.434372    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:04:00.047044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:04:00.047010    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:28.434589 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:04:28.434553    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:04:58.434954 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:04:58.434874    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:05:00.392423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:00.392368    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:27.005020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:27.004853    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:27.313947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:27.313864    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:28.434070 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:05:28.434031    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:05:28.530793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:28.530753    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:29.522671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.522637    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:29.785804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.785729    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"]
Oct 30 22:05:29.786034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786022    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" containerName="registry"
Oct 30 22:05:29.786075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786035    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" containerName="registry"
Oct 30 22:05:29.786075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786044    5028 cpu_manager.go:414] "containerMap: removing container" podUID="051a42ce-76bb-4e7e-a32c-82bff50cd38d" containerName="registry"
Oct 30 22:05:29.786075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786050    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051a42ce-76bb-4e7e-a32c-82bff50cd38d" containerName="registry"
Oct 30 22:05:29.786075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786064    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4b72179a-878b-4792-939e-6db6499f00bd" containerName="insights-gathering"
Oct 30 22:05:29.786075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786069    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b72179a-878b-4792-939e-6db6499f00bd" containerName="insights-gathering"
Oct 30 22:05:29.786225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786125    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f37686f-eba2-4b8a-bd8f-ca6c2fe38a59" containerName="registry"
Oct 30 22:05:29.786225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786133    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="051a42ce-76bb-4e7e-a32c-82bff50cd38d" containerName="registry"
Oct 30 22:05:29.786225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.786140    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b72179a-878b-4792-939e-6db6499f00bd" containerName="insights-gathering"
Oct 30 22:05:29.788343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.788323    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.817460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.817431    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jl6k\" (UniqueName: \"kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.817622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.817469    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.817622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.817495    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.838515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.838492    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 22:05:29.838515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.838512    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 22:05:29.886247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.886211    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7wgp9\""
Oct 30 22:05:29.887309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.887286    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"]
Oct 30 22:05:29.889929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.889908    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"]
Oct 30 22:05:29.892504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.892489    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:29.917935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.917904    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:29.918080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.917952    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jl6k\" (UniqueName: \"kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.918080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.917981    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.918080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.918001    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.918080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.918019    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:29.918296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.918084    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cvw\" (UniqueName: \"kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:29.918296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.918282    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.918434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.918414    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.940416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.940386    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:29.942806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.942785    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-d57hq\""
Oct 30 22:05:29.953673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.953650    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"]
Oct 30 22:05:29.977182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.977150    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jl6k\" (UniqueName: \"kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:29.992517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.992488    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"]
Oct 30 22:05:29.994986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:29.994968    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.019225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019193    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgv8\" (UniqueName: \"kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.019381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019250    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.019381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019321    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.019381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019351    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.019381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019367    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cvw\" (UniqueName: \"kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.019592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019434    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.019833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019812    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.019910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.019818    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.045164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.045096    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-fsj7d\""
Oct 30 22:05:30.051421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.051395    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"]
Oct 30 22:05:30.078316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.078283    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cvw\" (UniqueName: \"kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.090728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.090680    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"]
Oct 30 22:05:30.093282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.093264    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.100116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.100097    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:05:30.119773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.119742    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.119938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.119817    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.119938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.119852    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgv8\" (UniqueName: \"kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.119938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.119876    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.120108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.120031    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66cn\" (UniqueName: \"kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.120108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.120065    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.120204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.120161    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.120344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.120327    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.146302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.146266    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xbdd4\""
Oct 30 22:05:30.150971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.150738    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"]
Oct 30 22:05:30.179062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.179035    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgv8\" (UniqueName: \"kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.204782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.204752    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:05:30.221249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.221222    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66cn\" (UniqueName: \"kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.221381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.221288    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.221381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.221313    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.221726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.221705    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.221726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.221719    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.281171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.281096    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66cn\" (UniqueName: \"kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.308409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.308342    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:05:30.406604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.406568    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:05:30.444574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.444537    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"]
Oct 30 22:05:30.448348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.448325    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.467187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.467159    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"]
Oct 30 22:05:30.475164 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:30.475137    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144fbcc3_bd40_487a_a2f7_c4003df213b4.slice/crio-79d537af1c2a1af86dbb2c3583b422d69afb6810f330bffb88d460963344bd59 WatchSource:0}: Error finding container 79d537af1c2a1af86dbb2c3583b422d69afb6810f330bffb88d460963344bd59: Status 404 returned error can't find the container with id 79d537af1c2a1af86dbb2c3583b422d69afb6810f330bffb88d460963344bd59
Oct 30 22:05:30.498526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.498502    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-x8g8p\""
Oct 30 22:05:30.504449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.504424    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"]
Oct 30 22:05:30.523152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.523122    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9nn\" (UniqueName: \"kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.523547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.523172    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.523547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.523323    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.570764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.570740    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"]
Oct 30 22:05:30.572299 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:30.572270    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2019b8_49d6_49aa_803b_ba57aa1c8666.slice/crio-d6e5545ae091c16fa3d1461954a9be9ab07a9f02e0fb4a9ee310dd0923fd86d9 WatchSource:0}: Error finding container d6e5545ae091c16fa3d1461954a9be9ab07a9f02e0fb4a9ee310dd0923fd86d9: Status 404 returned error can't find the container with id d6e5545ae091c16fa3d1461954a9be9ab07a9f02e0fb4a9ee310dd0923fd86d9
Oct 30 22:05:30.624198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.624154    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.624364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.624222    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9nn\" (UniqueName: \"kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.624364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.624259    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.624567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.624544    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.624606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.624557    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.678641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.678614    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"]
Oct 30 22:05:30.680474 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:30.680445    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf018af50_65d9_4296_8228_2d855c5de80e.slice/crio-2172a439e47b458923ec5a29f3b6771f88bd74d11f86e7fdff1c037243be898b WatchSource:0}: Error finding container 2172a439e47b458923ec5a29f3b6771f88bd74d11f86e7fdff1c037243be898b: Status 404 returned error can't find the container with id 2172a439e47b458923ec5a29f3b6771f88bd74d11f86e7fdff1c037243be898b
Oct 30 22:05:30.685013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.684990    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9nn\" (UniqueName: \"kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.767960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.767931    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:05:30.773183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:30.773160    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"]
Oct 30 22:05:30.776769 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:30.776737    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa181c8_5382_4286_a90e_56a5023441a6.slice/crio-f54e2998547baaaa005f69141a31f7d69927b691a78fe2d98f0013efb238ca59 WatchSource:0}: Error finding container f54e2998547baaaa005f69141a31f7d69927b691a78fe2d98f0013efb238ca59: Status 404 returned error can't find the container with id f54e2998547baaaa005f69141a31f7d69927b691a78fe2d98f0013efb238ca59
Oct 30 22:05:31.133124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:31.132923    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"]
Oct 30 22:05:31.135521 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:31.135488    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dbe58b_affb_431a_9830_345e275354ed.slice/crio-e4b6b64af475aed7aa6d3125547d4c28c11908295d828d02cbcdb37e28e95069 WatchSource:0}: Error finding container e4b6b64af475aed7aa6d3125547d4c28c11908295d828d02cbcdb37e28e95069: Status 404 returned error can't find the container with id e4b6b64af475aed7aa6d3125547d4c28c11908295d828d02cbcdb37e28e95069
Oct 30 22:05:31.384395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:31.384301    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" event={"ID":"70dbe58b-affb-431a-9830-345e275354ed","Type":"ContainerStarted","Data":"e4b6b64af475aed7aa6d3125547d4c28c11908295d828d02cbcdb37e28e95069"}
Oct 30 22:05:31.385532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:31.385506    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq" event={"ID":"f018af50-65d9-4296-8228-2d855c5de80e","Type":"ContainerStarted","Data":"2172a439e47b458923ec5a29f3b6771f88bd74d11f86e7fdff1c037243be898b"}
Oct 30 22:05:31.386656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:31.386629    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" event={"ID":"144fbcc3-bd40-487a-a2f7-c4003df213b4","Type":"ContainerStarted","Data":"79d537af1c2a1af86dbb2c3583b422d69afb6810f330bffb88d460963344bd59"}
Oct 30 22:05:31.387748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:31.387712    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" event={"ID":"1aa181c8-5382-4286-a90e-56a5023441a6","Type":"ContainerStarted","Data":"f54e2998547baaaa005f69141a31f7d69927b691a78fe2d98f0013efb238ca59"}
Oct 30 22:05:31.388784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:31.388762    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt" event={"ID":"0c2019b8-49d6-49aa-803b-ba57aa1c8666","Type":"ContainerStarted","Data":"d6e5545ae091c16fa3d1461954a9be9ab07a9f02e0fb4a9ee310dd0923fd86d9"}
Oct 30 22:05:32.215935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.215896    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"]
Oct 30 22:05:32.220264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.220230    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.236442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.236418    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.236589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.236450    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwfh\" (UniqueName: \"kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.236802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.236605    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.274600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.274561    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"]
Oct 30 22:05:32.322919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.322887    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-sfh92\""
Oct 30 22:05:32.337785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.337750    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.337953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.337834    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.337953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.337864    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwfh\" (UniqueName: \"kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.338605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.338576    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.338904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.338856    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.398703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.398618    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwfh\" (UniqueName: \"kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.541349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.541035    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:05:32.954520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:32.954431    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"]
Oct 30 22:05:32.969220 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:32.969188    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7f7dbb_bb5e_4fd2_8bc0_0d92768d4ab0.slice/crio-2681cfff0e3de5408a555bc31f23944b02942055a5f8248aa7df14f02492792f WatchSource:0}: Error finding container 2681cfff0e3de5408a555bc31f23944b02942055a5f8248aa7df14f02492792f: Status 404 returned error can't find the container with id 2681cfff0e3de5408a555bc31f23944b02942055a5f8248aa7df14f02492792f
Oct 30 22:05:33.407990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:33.407880    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b" event={"ID":"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0","Type":"ContainerStarted","Data":"2681cfff0e3de5408a555bc31f23944b02942055a5f8248aa7df14f02492792f"}
Oct 30 22:05:35.037090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.036944    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:35.040492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.040464    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"]
Oct 30 22:05:35.045274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.044523    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.062539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.062505    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.062665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.062560    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9wq\" (UniqueName: \"kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.062752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.062705    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.142542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.142509    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7pwvt\""
Oct 30 22:05:35.144419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.144254    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"]
Oct 30 22:05:35.165222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.163864    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.165222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.163952    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.165222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.164001    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9wq\" (UniqueName: \"kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.165222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.164892    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.165222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.165147    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.195210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.195134    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:35.237580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.237514    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9wq\" (UniqueName: \"kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.370488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.370183    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:05:35.770045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:35.770000    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"]
Oct 30 22:05:35.788953 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:35.788917    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf7248d_6c02_45d8_9f43_10aa18ccc7ca.slice/crio-e06f00bfb51836368f29f06e1a07e6963f8d7ae699fb595d36a389d44dbb8e39 WatchSource:0}: Error finding container e06f00bfb51836368f29f06e1a07e6963f8d7ae699fb595d36a389d44dbb8e39: Status 404 returned error can't find the container with id e06f00bfb51836368f29f06e1a07e6963f8d7ae699fb595d36a389d44dbb8e39
Oct 30 22:05:36.348964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.348827    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.438386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.437773    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7" event={"ID":"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca","Type":"ContainerStarted","Data":"e06f00bfb51836368f29f06e1a07e6963f8d7ae699fb595d36a389d44dbb8e39"}
Oct 30 22:05:36.529241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.527807    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"]
Oct 30 22:05:36.533946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.533503    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.588123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.588082    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-gsnvr\""
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590034    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cca595-c64b-4d4c-a9be-b955ec62c585-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590095    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590128    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590168    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590192    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590242    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590314    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbn2x\" (UniqueName: \"kubernetes.io/projected/e5cca595-c64b-4d4c-a9be-b955ec62c585-kube-api-access-xbn2x\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590348    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.590484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.590365    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.617020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.616992    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:05:36.619315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.619290    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"]
Oct 30 22:05:36.622043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.621892    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691486    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691540    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbn2x\" (UniqueName: \"kubernetes.io/projected/e5cca595-c64b-4d4c-a9be-b955ec62c585-kube-api-access-xbn2x\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691574    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691603    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691670    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cca595-c64b-4d4c-a9be-b955ec62c585-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691737    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691787    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691828    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.691879    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.692627    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.693557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.693520    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.694305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.693913    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.694305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.694021    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.694626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.694540    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cca595-c64b-4d4c-a9be-b955ec62c585-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.695572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.695550    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.697153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.697129    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.752900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.752866    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cca595-c64b-4d4c-a9be-b955ec62c585-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.753293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.753257    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbn2x\" (UniqueName: \"kubernetes.io/projected/e5cca595-c64b-4d4c-a9be-b955ec62c585-kube-api-access-xbn2x\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk\" (UID: \"e5cca595-c64b-4d4c-a9be-b955ec62c585\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:36.771866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.771796    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.856999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:36.856597    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:37.301736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:37.301593    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"]
Oct 30 22:05:37.450127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:37.450088    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk" event={"ID":"e5cca595-c64b-4d4c-a9be-b955ec62c585","Type":"ContainerStarted","Data":"038b05e7239ddee919f023a2fcaccdc0e289459bbf23871fa73ba20238ba1af8"}
Oct 30 22:05:38.013813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.011293    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"]
Oct 30 22:05:38.016021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.015991    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.071607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.071563    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"]
Oct 30 22:05:38.114896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.114854    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.115077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.114908    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf64m\" (UniqueName: \"kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.115151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.115081    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.215775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.215738    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.215959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.215788    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf64m\" (UniqueName: \"kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.215959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.215876    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.216489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.216463    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.216864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.216840    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.295329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.295255    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf64m\" (UniqueName: \"kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.339407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.339359    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:05:38.746678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:38.746645    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"]
Oct 30 22:05:38.750441 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:05:38.750409    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085fa64e_330e_4148_bbc7_d8a5eec1c695.slice/crio-3a7bed1642f3738ae5313dd94a6074d9f8ff8c6e21e83f0bf792810fdc93b39a WatchSource:0}: Error finding container 3a7bed1642f3738ae5313dd94a6074d9f8ff8c6e21e83f0bf792810fdc93b39a: Status 404 returned error can't find the container with id 3a7bed1642f3738ae5313dd94a6074d9f8ff8c6e21e83f0bf792810fdc93b39a
Oct 30 22:05:39.470452 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:39.470412    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8" event={"ID":"085fa64e-330e-4148-bbc7-d8a5eec1c695","Type":"ContainerStarted","Data":"3a7bed1642f3738ae5313dd94a6074d9f8ff8c6e21e83f0bf792810fdc93b39a"}
Oct 30 22:05:40.551989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:40.551920    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:46.533881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:46.533801    5028 generic.go:356] "Generic (PLEG): container finished" podID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerID="c9987df07255ee490a46df1a7a5dc7359b714117d50f097097f1c1ce99c95a1f" exitCode=0
Oct 30 22:05:46.534277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:46.534155    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt" event={"ID":"0c2019b8-49d6-49aa-803b-ba57aa1c8666","Type":"ContainerDied","Data":"c9987df07255ee490a46df1a7a5dc7359b714117d50f097097f1c1ce99c95a1f"}
Oct 30 22:05:46.536934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:46.536906    5028 generic.go:356] "Generic (PLEG): container finished" podID="f018af50-65d9-4296-8228-2d855c5de80e" containerID="dd9e1a7ca73a3250964b6c33bb4cc74c618f317b21d29191a1bda8dbbe4bd1f1" exitCode=0
Oct 30 22:05:46.537039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:46.536943    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq" event={"ID":"f018af50-65d9-4296-8228-2d855c5de80e","Type":"ContainerDied","Data":"dd9e1a7ca73a3250964b6c33bb4cc74c618f317b21d29191a1bda8dbbe4bd1f1"}
Oct 30 22:05:47.051594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.051555    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:47.051734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.051644    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:47.051734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.051698    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:47.543486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.543444    5028 generic.go:356] "Generic (PLEG): container finished" podID="70dbe58b-affb-431a-9830-345e275354ed" containerID="1ef59df710dbe47ad93d40fcd265678ac0d9c1c6cb8df4f9339e4e26122bac2c" exitCode=0
Oct 30 22:05:47.543967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.543541    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" event={"ID":"70dbe58b-affb-431a-9830-345e275354ed","Type":"ContainerDied","Data":"1ef59df710dbe47ad93d40fcd265678ac0d9c1c6cb8df4f9339e4e26122bac2c"}
Oct 30 22:05:47.546757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.546725    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk" event={"ID":"e5cca595-c64b-4d4c-a9be-b955ec62c585","Type":"ContainerStarted","Data":"eda8fb0764320f6f6faa6cf8d138805c530b31d0e16f2e0162b1ef829e70b9a9"}
Oct 30 22:05:47.548834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.548669    5028 generic.go:356] "Generic (PLEG): container finished" podID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerID="64911d57c57c835f4c406651ed1e71cfe74ca1a1e8811a1bb6e42695f5aa538a" exitCode=0
Oct 30 22:05:47.548834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.548777    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" event={"ID":"144fbcc3-bd40-487a-a2f7-c4003df213b4","Type":"ContainerDied","Data":"64911d57c57c835f4c406651ed1e71cfe74ca1a1e8811a1bb6e42695f5aa538a"}
Oct 30 22:05:47.551180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.551156    5028 generic.go:356] "Generic (PLEG): container finished" podID="1aa181c8-5382-4286-a90e-56a5023441a6" containerID="0cbd23cee025d82681717f2867802498d811da0c15de26e916f1798645f8f832" exitCode=0
Oct 30 22:05:47.551349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.551223    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" event={"ID":"1aa181c8-5382-4286-a90e-56a5023441a6","Type":"ContainerDied","Data":"0cbd23cee025d82681717f2867802498d811da0c15de26e916f1798645f8f832"}
Oct 30 22:05:47.553716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.553673    5028 generic.go:356] "Generic (PLEG): container finished" podID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerID="6554734fd006746e9ff0192a2a9cfc98db4ff21c97550cdab850f4deaba74cf6" exitCode=0
Oct 30 22:05:47.553795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.553727    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7" event={"ID":"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca","Type":"ContainerDied","Data":"6554734fd006746e9ff0192a2a9cfc98db4ff21c97550cdab850f4deaba74cf6"}
Oct 30 22:05:47.857213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.857133    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:47.997183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:47.997136    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk" podStartSLOduration=2.257644705 podStartE2EDuration="11.997122602s" totalImagesPullingTime="9.739477897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:47.996821092 +0000 UTC m=+278.846334069" watchObservedRunningTime="2026-10-30 22:05:47.997122602 +0000 UTC m=+278.846635576"
Oct 30 22:05:48.559363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:48.559335    5028 generic.go:356] "Generic (PLEG): container finished" podID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerID="12cb90bbcce4048ef869dfb91f7591c3e5132c0c59b406f3b083ec4185d9a995" exitCode=0
Oct 30 22:05:48.559740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:48.559412    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8" event={"ID":"085fa64e-330e-4148-bbc7-d8a5eec1c695","Type":"ContainerDied","Data":"12cb90bbcce4048ef869dfb91f7591c3e5132c0c59b406f3b083ec4185d9a995"}
Oct 30 22:05:48.560904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:48.560884    5028 generic.go:356] "Generic (PLEG): container finished" podID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerID="f60331505e073a649ad0f116fb4d395ee2524b3da1d42452e1030c1d0175e551" exitCode=0
Oct 30 22:05:48.561000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:48.560964    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b" event={"ID":"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0","Type":"ContainerDied","Data":"f60331505e073a649ad0f116fb4d395ee2524b3da1d42452e1030c1d0175e551"}
Oct 30 22:05:48.861598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:48.861569    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:49.567538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.567444    5028 generic.go:356] "Generic (PLEG): container finished" podID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerID="3dd2364eaaef740344c1eb05443c8740d6ffc3e3ba4f8a777b90be9c2b840488" exitCode=0
Oct 30 22:05:49.567538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.567489    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt" event={"ID":"0c2019b8-49d6-49aa-803b-ba57aa1c8666","Type":"ContainerDied","Data":"3dd2364eaaef740344c1eb05443c8740d6ffc3e3ba4f8a777b90be9c2b840488"}
Oct 30 22:05:49.569495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.569477    5028 generic.go:356] "Generic (PLEG): container finished" podID="f018af50-65d9-4296-8228-2d855c5de80e" containerID="e414bd574f187716d28435c4d3570e1c9efc16cda327eb97defc6571a6b28bc1" exitCode=0
Oct 30 22:05:49.569559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.569542    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq" event={"ID":"f018af50-65d9-4296-8228-2d855c5de80e","Type":"ContainerDied","Data":"e414bd574f187716d28435c4d3570e1c9efc16cda327eb97defc6571a6b28bc1"}
Oct 30 22:05:49.571444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.571426    5028 generic.go:356] "Generic (PLEG): container finished" podID="1aa181c8-5382-4286-a90e-56a5023441a6" containerID="8404274a338a689cfa710aa49c52bd385306ec7cf4b424c4a00f2b97528f9a5b" exitCode=0
Oct 30 22:05:49.571534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.571501    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" event={"ID":"1aa181c8-5382-4286-a90e-56a5023441a6","Type":"ContainerDied","Data":"8404274a338a689cfa710aa49c52bd385306ec7cf4b424c4a00f2b97528f9a5b"}
Oct 30 22:05:49.573434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.573413    5028 generic.go:356] "Generic (PLEG): container finished" podID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerID="9c7e5efde8a53aa53f2470bc0bec44db28751f853c135c7428e88b2a93431e9a" exitCode=0
Oct 30 22:05:49.573525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.573474    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7" event={"ID":"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca","Type":"ContainerDied","Data":"9c7e5efde8a53aa53f2470bc0bec44db28751f853c135c7428e88b2a93431e9a"}
Oct 30 22:05:49.573743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.573721    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:49.574790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:49.574771    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk"
Oct 30 22:05:50.337252 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:05:50.337218    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085fa64e_330e_4148_bbc7_d8a5eec1c695.slice/crio-conmon-347b6e4bc4a65e475ce663cd8815177ae11fead522b715f7a106d0491639314f.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:05:50.582423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:50.582380    5028 generic.go:356] "Generic (PLEG): container finished" podID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerID="478d40551b61e4446eba1775f1e3b4e1d7328c3c21406239a05c17156daf16b3" exitCode=0
Oct 30 22:05:50.583211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:50.583139    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" event={"ID":"144fbcc3-bd40-487a-a2f7-c4003df213b4","Type":"ContainerDied","Data":"478d40551b61e4446eba1775f1e3b4e1d7328c3c21406239a05c17156daf16b3"}
Oct 30 22:05:50.588931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:50.587140    5028 generic.go:356] "Generic (PLEG): container finished" podID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerID="347b6e4bc4a65e475ce663cd8815177ae11fead522b715f7a106d0491639314f" exitCode=0
Oct 30 22:05:50.588931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:50.588491    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8" event={"ID":"085fa64e-330e-4148-bbc7-d8a5eec1c695","Type":"ContainerDied","Data":"347b6e4bc4a65e475ce663cd8815177ae11fead522b715f7a106d0491639314f"}
Oct 30 22:05:54.634666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:54.634462    5028 generic.go:356] "Generic (PLEG): container finished" podID="70dbe58b-affb-431a-9830-345e275354ed" containerID="5ad5ebd407c624e141bb508b101d22e2e959c1b64b7dc4239cbf573187d2a9d3" exitCode=0
Oct 30 22:05:54.634666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:54.634573    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" event={"ID":"70dbe58b-affb-431a-9830-345e275354ed","Type":"ContainerDied","Data":"5ad5ebd407c624e141bb508b101d22e2e959c1b64b7dc4239cbf573187d2a9d3"}
Oct 30 22:05:55.652843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:55.652705    5028 generic.go:356] "Generic (PLEG): container finished" podID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerID="f86fb140850ec4f49852dc47b0956d62c4fee0af9485ba83dbcf2e235d696735" exitCode=0
Oct 30 22:05:55.653357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:05:55.652900    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b" event={"ID":"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0","Type":"ContainerDied","Data":"f86fb140850ec4f49852dc47b0956d62c4fee0af9485ba83dbcf2e235d696735"}
Oct 30 22:05:58.472561 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:05:58.472520    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:06:00.754728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:00.754648    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:06.761797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:06.761758    5028 generic.go:356] "Generic (PLEG): container finished" podID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerID="0b5599fb02454944fbf173b2b400d2400b4e9a30e520dfc35df114351cbfe3fa" exitCode=0
Oct 30 22:06:06.762279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:06.761817    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt" event={"ID":"0c2019b8-49d6-49aa-803b-ba57aa1c8666","Type":"ContainerDied","Data":"0b5599fb02454944fbf173b2b400d2400b4e9a30e520dfc35df114351cbfe3fa"}
Oct 30 22:06:07.770451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.769892    5028 generic.go:356] "Generic (PLEG): container finished" podID="f018af50-65d9-4296-8228-2d855c5de80e" containerID="733f37d73fe199bcda524f3ebcc58e3e6a4b78ad2afe825f04422e880f66f4e5" exitCode=0
Oct 30 22:06:07.770451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.769999    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq" event={"ID":"f018af50-65d9-4296-8228-2d855c5de80e","Type":"ContainerDied","Data":"733f37d73fe199bcda524f3ebcc58e3e6a4b78ad2afe825f04422e880f66f4e5"}
Oct 30 22:06:07.775466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.774089    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" event={"ID":"144fbcc3-bd40-487a-a2f7-c4003df213b4","Type":"ContainerStarted","Data":"17eb85dd6dbc4045479a40e512013d06f11f2d9f854f8c1ae196dcd5b9465b1c"}
Oct 30 22:06:07.777105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.777079    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" event={"ID":"1aa181c8-5382-4286-a90e-56a5023441a6","Type":"ContainerStarted","Data":"cc9154e8f92363209b03d7f4ceec9422010774445bf1393a73219e83367f2610"}
Oct 30 22:06:07.779391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.779364    5028 generic.go:356] "Generic (PLEG): container finished" podID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerID="fc532ad45c686e7ccd6cd923a64c91a419b5aa860f5a0431c6931ae0f37f8d47" exitCode=0
Oct 30 22:06:07.779491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.779449    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7" event={"ID":"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca","Type":"ContainerDied","Data":"fc532ad45c686e7ccd6cd923a64c91a419b5aa860f5a0431c6931ae0f37f8d47"}
Oct 30 22:06:07.781765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.781741    5028 generic.go:356] "Generic (PLEG): container finished" podID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerID="104528e14afb382bc8962de221cc10d0c40a5984ae9dd19b2a2b0a1f2bdd9801" exitCode=0
Oct 30 22:06:07.781859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.781832    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b" event={"ID":"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0","Type":"ContainerDied","Data":"104528e14afb382bc8962de221cc10d0c40a5984ae9dd19b2a2b0a1f2bdd9801"}
Oct 30 22:06:07.933753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.933719    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:06:07.995820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:07.995759    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" podStartSLOduration=3.791503165 podStartE2EDuration="38.995744892s" totalImagesPullingTime="35.204241727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:07.995548631 +0000 UTC m=+298.845061606" watchObservedRunningTime="2026-10-30 22:06:07.995744892 +0000 UTC m=+298.845257861"
Oct 30 22:06:08.010861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.010829    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw\" (UniqueName: \"kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw\") pod \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") "
Oct 30 22:06:08.011027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.010902    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle\") pod \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") "
Oct 30 22:06:08.011027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.010999    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util\") pod \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\" (UID: \"0c2019b8-49d6-49aa-803b-ba57aa1c8666\") "
Oct 30 22:06:08.011848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.011816    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle" pod "0c2019b8-49d6-49aa-803b-ba57aa1c8666" (UID: "0c2019b8-49d6-49aa-803b-ba57aa1c8666"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:08.014197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.014167    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw" pod "0c2019b8-49d6-49aa-803b-ba57aa1c8666" (UID: "0c2019b8-49d6-49aa-803b-ba57aa1c8666"). InnerVolumeSpecName "kube-api-access-q6cvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:08.017445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.017416    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util" pod "0c2019b8-49d6-49aa-803b-ba57aa1c8666" (UID: "0c2019b8-49d6-49aa-803b-ba57aa1c8666"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:08.112037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.112003    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:08.112037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.112036    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cvw\" (UniqueName: \"kubernetes.io/projected/0c2019b8-49d6-49aa-803b-ba57aa1c8666-kube-api-access-q6cvw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:08.112236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.112046    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c2019b8-49d6-49aa-803b-ba57aa1c8666-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:08.226117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.226056    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" podStartSLOduration=2.589198003 podStartE2EDuration="38.226040451s" totalImagesPullingTime="35.636842448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:08.22481451 +0000 UTC m=+299.074327483" watchObservedRunningTime="2026-10-30 22:06:08.226040451 +0000 UTC m=+299.075553427"
Oct 30 22:06:08.789379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.789339    5028 generic.go:356] "Generic (PLEG): container finished" podID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerID="17eb85dd6dbc4045479a40e512013d06f11f2d9f854f8c1ae196dcd5b9465b1c" exitCode=0
Oct 30 22:06:08.789868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.789426    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" event={"ID":"144fbcc3-bd40-487a-a2f7-c4003df213b4","Type":"ContainerDied","Data":"17eb85dd6dbc4045479a40e512013d06f11f2d9f854f8c1ae196dcd5b9465b1c"}
Oct 30 22:06:08.791976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.791949    5028 generic.go:356] "Generic (PLEG): container finished" podID="1aa181c8-5382-4286-a90e-56a5023441a6" containerID="cc9154e8f92363209b03d7f4ceec9422010774445bf1393a73219e83367f2610" exitCode=0
Oct 30 22:06:08.792109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.792023    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" event={"ID":"1aa181c8-5382-4286-a90e-56a5023441a6","Type":"ContainerDied","Data":"cc9154e8f92363209b03d7f4ceec9422010774445bf1393a73219e83367f2610"}
Oct 30 22:06:08.794507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.794482    5028 generic.go:356] "Generic (PLEG): container finished" podID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerID="71dfde65be1ca2a51d4196a0fe32208d2df878c9d3888a1b5422e4fe7e75532b" exitCode=0
Oct 30 22:06:08.794605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.794555    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8" event={"ID":"085fa64e-330e-4148-bbc7-d8a5eec1c695","Type":"ContainerDied","Data":"71dfde65be1ca2a51d4196a0fe32208d2df878c9d3888a1b5422e4fe7e75532b"}
Oct 30 22:06:08.796781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.796757    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt" event={"ID":"0c2019b8-49d6-49aa-803b-ba57aa1c8666","Type":"ContainerDied","Data":"d6e5545ae091c16fa3d1461954a9be9ab07a9f02e0fb4a9ee310dd0923fd86d9"}
Oct 30 22:06:08.796876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.796800    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e5545ae091c16fa3d1461954a9be9ab07a9f02e0fb4a9ee310dd0923fd86d9"
Oct 30 22:06:08.799511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.797243    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c7fvt"
Oct 30 22:06:08.981092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:08.981059    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:06:09.009850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.009824    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:06:09.013772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.013752    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:06:09.119618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119590    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq\" (UniqueName: \"kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq\") pod \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") "
Oct 30 22:06:09.119757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119632    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8\" (UniqueName: \"kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8\") pod \"f018af50-65d9-4296-8228-2d855c5de80e\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") "
Oct 30 22:06:09.119757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119653    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle\") pod \"f018af50-65d9-4296-8228-2d855c5de80e\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") "
Oct 30 22:06:09.119757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119712    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util\") pod \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") "
Oct 30 22:06:09.119757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119749    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle\") pod \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\" (UID: \"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca\") "
Oct 30 22:06:09.119971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119783    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util\") pod \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") "
Oct 30 22:06:09.119971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119807    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh\" (UniqueName: \"kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh\") pod \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") "
Oct 30 22:06:09.119971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119840    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle\") pod \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\" (UID: \"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0\") "
Oct 30 22:06:09.119971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.119933    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util\") pod \"f018af50-65d9-4296-8228-2d855c5de80e\" (UID: \"f018af50-65d9-4296-8228-2d855c5de80e\") "
Oct 30 22:06:09.120719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.120666    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle" pod "fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" (UID: "fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:09.120821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.120769    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle" pod "f018af50-65d9-4296-8228-2d855c5de80e" (UID: "f018af50-65d9-4296-8228-2d855c5de80e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:09.120821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.120775    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle" pod "2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" (UID: "2cf7248d-6c02-45d8-9f43-10aa18ccc7ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:09.122320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.122291    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8" pod "f018af50-65d9-4296-8228-2d855c5de80e" (UID: "f018af50-65d9-4296-8228-2d855c5de80e"). InnerVolumeSpecName "kube-api-access-qjgv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:09.122526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.122507    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq" pod "2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" (UID: "2cf7248d-6c02-45d8-9f43-10aa18ccc7ca"). InnerVolumeSpecName "kube-api-access-lm9wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:09.122854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.122829    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh" pod "fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" (UID: "fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0"). InnerVolumeSpecName "kube-api-access-6dwfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:09.126804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.126784    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util" pod "fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" (UID: "fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:09.127301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.127280    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util" pod "f018af50-65d9-4296-8228-2d855c5de80e" (UID: "f018af50-65d9-4296-8228-2d855c5de80e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:09.128199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.128173    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util" pod "2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" (UID: "2cf7248d-6c02-45d8-9f43-10aa18ccc7ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:09.220534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220500    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220529    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220539    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwfh\" (UniqueName: \"kubernetes.io/projected/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-kube-api-access-6dwfh\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220549    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220557    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220566    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9wq\" (UniqueName: \"kubernetes.io/projected/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-kube-api-access-lm9wq\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220574    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgv8\" (UniqueName: \"kubernetes.io/projected/f018af50-65d9-4296-8228-2d855c5de80e-kube-api-access-qjgv8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220583    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f018af50-65d9-4296-8228-2d855c5de80e-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.220790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.220590    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf7248d-6c02-45d8-9f43-10aa18ccc7ca-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.318156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.318120    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:06:09.319221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.319169    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:06:09.803937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.803902    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" event={"ID":"70dbe58b-affb-431a-9830-345e275354ed","Type":"ContainerStarted","Data":"cce6dc7e35b3bfe2ad1525b9bb8861de6ef7d23d257d06df423f1d4190981dc1"}
Oct 30 22:06:09.805862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.805839    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq"
Oct 30 22:06:09.805988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.805845    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aj6vqq" event={"ID":"f018af50-65d9-4296-8228-2d855c5de80e","Type":"ContainerDied","Data":"2172a439e47b458923ec5a29f3b6771f88bd74d11f86e7fdff1c037243be898b"}
Oct 30 22:06:09.805988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.805943    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2172a439e47b458923ec5a29f3b6771f88bd74d11f86e7fdff1c037243be898b"
Oct 30 22:06:09.808046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.808025    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7"
Oct 30 22:06:09.808046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.808043    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716mhh7" event={"ID":"2cf7248d-6c02-45d8-9f43-10aa18ccc7ca","Type":"ContainerDied","Data":"e06f00bfb51836368f29f06e1a07e6963f8d7ae699fb595d36a389d44dbb8e39"}
Oct 30 22:06:09.808263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.808091    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e06f00bfb51836368f29f06e1a07e6963f8d7ae699fb595d36a389d44dbb8e39"
Oct 30 22:06:09.810587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.810556    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b" event={"ID":"fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0","Type":"ContainerDied","Data":"2681cfff0e3de5408a555bc31f23944b02942055a5f8248aa7df14f02492792f"}
Oct 30 22:06:09.810709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.810608    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2681cfff0e3de5408a555bc31f23944b02942055a5f8248aa7df14f02492792f"
Oct 30 22:06:09.811398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.811056    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wsw6b"
Oct 30 22:06:09.920123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.920028    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" podStartSLOduration=3.096798395 podStartE2EDuration="39.920011163s" totalImagesPullingTime="36.823212768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:09.919327351 +0000 UTC m=+300.768840324" watchObservedRunningTime="2026-10-30 22:06:09.920011163 +0000 UTC m=+300.769524135"
Oct 30 22:06:09.956888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.956858    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:06:09.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.994738    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:06:09.997714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:09.997680    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:06:10.029837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.029798    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m\" (UniqueName: \"kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m\") pod \"085fa64e-330e-4148-bbc7-d8a5eec1c695\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") "
Oct 30 22:06:10.029976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.029893    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util\") pod \"085fa64e-330e-4148-bbc7-d8a5eec1c695\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") "
Oct 30 22:06:10.029976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.029928    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle\") pod \"085fa64e-330e-4148-bbc7-d8a5eec1c695\" (UID: \"085fa64e-330e-4148-bbc7-d8a5eec1c695\") "
Oct 30 22:06:10.030411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.030386    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle" pod "085fa64e-330e-4148-bbc7-d8a5eec1c695" (UID: "085fa64e-330e-4148-bbc7-d8a5eec1c695"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:10.032020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.031993    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m" pod "085fa64e-330e-4148-bbc7-d8a5eec1c695" (UID: "085fa64e-330e-4148-bbc7-d8a5eec1c695"). InnerVolumeSpecName "kube-api-access-lf64m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:10.035716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.035676    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util" pod "085fa64e-330e-4148-bbc7-d8a5eec1c695" (UID: "085fa64e-330e-4148-bbc7-d8a5eec1c695"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:10.130557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130472    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle\") pod \"144fbcc3-bd40-487a-a2f7-c4003df213b4\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") "
Oct 30 22:06:10.130557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130521    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k\" (UniqueName: \"kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k\") pod \"144fbcc3-bd40-487a-a2f7-c4003df213b4\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") "
Oct 30 22:06:10.130815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130568    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util\") pod \"144fbcc3-bd40-487a-a2f7-c4003df213b4\" (UID: \"144fbcc3-bd40-487a-a2f7-c4003df213b4\") "
Oct 30 22:06:10.130815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130609    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle\") pod \"1aa181c8-5382-4286-a90e-56a5023441a6\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") "
Oct 30 22:06:10.130815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130637    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn\" (UniqueName: \"kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn\") pod \"1aa181c8-5382-4286-a90e-56a5023441a6\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") "
Oct 30 22:06:10.130815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130658    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util\") pod \"1aa181c8-5382-4286-a90e-56a5023441a6\" (UID: \"1aa181c8-5382-4286-a90e-56a5023441a6\") "
Oct 30 22:06:10.131026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130928    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf64m\" (UniqueName: \"kubernetes.io/projected/085fa64e-330e-4148-bbc7-d8a5eec1c695-kube-api-access-lf64m\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.131026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130948    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.131026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.130963    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/085fa64e-330e-4148-bbc7-d8a5eec1c695-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.131218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.131182    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle" pod "144fbcc3-bd40-487a-a2f7-c4003df213b4" (UID: "144fbcc3-bd40-487a-a2f7-c4003df213b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:10.131330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.131302    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle" pod "1aa181c8-5382-4286-a90e-56a5023441a6" (UID: "1aa181c8-5382-4286-a90e-56a5023441a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:10.133376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.133353    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn" pod "1aa181c8-5382-4286-a90e-56a5023441a6" (UID: "1aa181c8-5382-4286-a90e-56a5023441a6"). InnerVolumeSpecName "kube-api-access-v66cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:10.133490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.133422    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k" pod "144fbcc3-bd40-487a-a2f7-c4003df213b4" (UID: "144fbcc3-bd40-487a-a2f7-c4003df213b4"). InnerVolumeSpecName "kube-api-access-8jl6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:10.135750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.135725    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util" pod "144fbcc3-bd40-487a-a2f7-c4003df213b4" (UID: "144fbcc3-bd40-487a-a2f7-c4003df213b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:10.136399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.136375    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util" pod "1aa181c8-5382-4286-a90e-56a5023441a6" (UID: "1aa181c8-5382-4286-a90e-56a5023441a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:10.232352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.232310    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.232352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.232345    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66cn\" (UniqueName: \"kubernetes.io/projected/1aa181c8-5382-4286-a90e-56a5023441a6-kube-api-access-v66cn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.232352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.232356    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1aa181c8-5382-4286-a90e-56a5023441a6-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.232585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.232365    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.232585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.232374    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jl6k\" (UniqueName: \"kubernetes.io/projected/144fbcc3-bd40-487a-a2f7-c4003df213b4-kube-api-access-8jl6k\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.232585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.232381    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/144fbcc3-bd40-487a-a2f7-c4003df213b4-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.817339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.817306    5028 generic.go:356] "Generic (PLEG): container finished" podID="70dbe58b-affb-431a-9830-345e275354ed" containerID="cce6dc7e35b3bfe2ad1525b9bb8861de6ef7d23d257d06df423f1d4190981dc1" exitCode=0
Oct 30 22:06:10.817798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.817398    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" event={"ID":"70dbe58b-affb-431a-9830-345e275354ed","Type":"ContainerDied","Data":"cce6dc7e35b3bfe2ad1525b9bb8861de6ef7d23d257d06df423f1d4190981dc1"}
Oct 30 22:06:10.819356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.819332    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8" event={"ID":"144fbcc3-bd40-487a-a2f7-c4003df213b4","Type":"ContainerDied","Data":"79d537af1c2a1af86dbb2c3583b422d69afb6810f330bffb88d460963344bd59"}
Oct 30 22:06:10.819464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.819376    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d537af1c2a1af86dbb2c3583b422d69afb6810f330bffb88d460963344bd59"
Oct 30 22:06:10.819464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.819335    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c56v8"
Oct 30 22:06:10.821228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.821208    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v"
Oct 30 22:06:10.821315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.821207    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck6n2v" event={"ID":"1aa181c8-5382-4286-a90e-56a5023441a6","Type":"ContainerDied","Data":"f54e2998547baaaa005f69141a31f7d69927b691a78fe2d98f0013efb238ca59"}
Oct 30 22:06:10.821315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.821304    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54e2998547baaaa005f69141a31f7d69927b691a78fe2d98f0013efb238ca59"
Oct 30 22:06:10.823450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.823418    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8" event={"ID":"085fa64e-330e-4148-bbc7-d8a5eec1c695","Type":"ContainerDied","Data":"3a7bed1642f3738ae5313dd94a6074d9f8ff8c6e21e83f0bf792810fdc93b39a"}
Oct 30 22:06:10.823533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.823461    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a7bed1642f3738ae5313dd94a6074d9f8ff8c6e21e83f0bf792810fdc93b39a"
Oct 30 22:06:10.823533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:10.823434    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hpwc8"
Oct 30 22:06:11.952884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:11.952859    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:06:12.049957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.049927    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn\" (UniqueName: \"kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn\") pod \"70dbe58b-affb-431a-9830-345e275354ed\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") "
Oct 30 22:06:12.050138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.049965    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util\") pod \"70dbe58b-affb-431a-9830-345e275354ed\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") "
Oct 30 22:06:12.050138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.050030    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle\") pod \"70dbe58b-affb-431a-9830-345e275354ed\" (UID: \"70dbe58b-affb-431a-9830-345e275354ed\") "
Oct 30 22:06:12.050733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.050680    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle" pod "70dbe58b-affb-431a-9830-345e275354ed" (UID: "70dbe58b-affb-431a-9830-345e275354ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:12.052071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.052046    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn" pod "70dbe58b-affb-431a-9830-345e275354ed" (UID: "70dbe58b-affb-431a-9830-345e275354ed"). InnerVolumeSpecName "kube-api-access-kr9nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:12.055650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.055628    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util" pod "70dbe58b-affb-431a-9830-345e275354ed" (UID: "70dbe58b-affb-431a-9830-345e275354ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:12.150951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.150869    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr9nn\" (UniqueName: \"kubernetes.io/projected/70dbe58b-affb-431a-9830-345e275354ed-kube-api-access-kr9nn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:12.150951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.150900    5028 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-util\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:12.150951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.150909    5028 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70dbe58b-affb-431a-9830-345e275354ed-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:12.498402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.498370    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:12.718943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.718906    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:12.835345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.835268    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w"
Oct 30 22:06:12.835345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.835279    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clc46w" event={"ID":"70dbe58b-affb-431a-9830-345e275354ed","Type":"ContainerDied","Data":"e4b6b64af475aed7aa6d3125547d4c28c11908295d828d02cbcdb37e28e95069"}
Oct 30 22:06:12.835345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.835323    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4b6b64af475aed7aa6d3125547d4c28c11908295d828d02cbcdb37e28e95069"
Oct 30 22:06:12.889565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:12.889533    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:13.387990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.387955    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:13.548363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548327    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"]
Oct 30 22:06:13.548770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548751    5028 cpu_manager.go:414] "containerMap: removing container" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="util"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548776    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="util"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548795    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="pull"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548801    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="pull"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548814    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="extract"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548821    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="extract"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548833    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="pull"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548840    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="pull"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548852    5028 cpu_manager.go:414] "containerMap: removing container" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="pull"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548860    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="pull"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548871    5028 cpu_manager.go:414] "containerMap: removing container" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="extract"
Oct 30 22:06:13.548880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548878    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548890    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548898    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548916    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548923    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548933    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548940    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548950    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548957    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548967    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548974    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548985    5028 cpu_manager.go:414] "containerMap: removing container" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.548992    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549006    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549013    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549022    5028 cpu_manager.go:414] "containerMap: removing container" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549030    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549043    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549050    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549062    5028 cpu_manager.go:414] "containerMap: removing container" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549069    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549078    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549085    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549094    5028 cpu_manager.go:414] "containerMap: removing container" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549102    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549128    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549136    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549148    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549156    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549168    5028 cpu_manager.go:414] "containerMap: removing container" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549175    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549183    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549190    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="util"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549200    5028 cpu_manager.go:414] "containerMap: removing container" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549207    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549222    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549228    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="pull"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549312    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="085fa64e-330e-4148-bbc7-d8a5eec1c695" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549324    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2019b8-49d6-49aa-803b-ba57aa1c8666" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549334    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf7248d-6c02-45d8-9f43-10aa18ccc7ca" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549341    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="144fbcc3-bd40-487a-a2f7-c4003df213b4" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549351    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa181c8-5382-4286-a90e-56a5023441a6" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549359    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dbe58b-affb-431a-9830-345e275354ed" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549369    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7f7dbb-bb5e-4fd2-8bc0-0d92768d4ab0" containerName="extract"
Oct 30 22:06:13.549411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.549380    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f018af50-65d9-4296-8228-2d855c5de80e" containerName="extract"
Oct 30 22:06:13.551925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.551905    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.562148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.562121    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:13.606111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.606080    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 22:06:13.606111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.606104    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8hr44\""
Oct 30 22:06:13.606309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.606091    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 22:06:13.606309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.606088    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 22:06:13.650264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.650189    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 22:06:13.650264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.650189    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 22:06:13.664247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.664220    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bmc\" (UniqueName: \"kubernetes.io/projected/358e1f33-8740-4009-8e8a-4c99647ac0b0-kube-api-access-94bmc\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.664302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.664263    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.664302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.664290    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/358e1f33-8740-4009-8e8a-4c99647ac0b0-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.664426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.664413    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.664462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.664442    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.664494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.664461    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/358e1f33-8740-4009-8e8a-4c99647ac0b0-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.716039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.716009    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:13.765640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.765607    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.765863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.765647    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.765863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.765668    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/358e1f33-8740-4009-8e8a-4c99647ac0b0-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.765863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.765746    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bmc\" (UniqueName: \"kubernetes.io/projected/358e1f33-8740-4009-8e8a-4c99647ac0b0-kube-api-access-94bmc\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.765863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.765786    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.766081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.765879    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/358e1f33-8740-4009-8e8a-4c99647ac0b0-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.766145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.766119    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/358e1f33-8740-4009-8e8a-4c99647ac0b0-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.768191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.768165    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/358e1f33-8740-4009-8e8a-4c99647ac0b0-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.768423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.768402    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.768486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.768436    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.768486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.768453    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/358e1f33-8740-4009-8e8a-4c99647ac0b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.812790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.812757    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"]
Oct 30 22:06:13.830260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.830225    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bmc\" (UniqueName: \"kubernetes.io/projected/358e1f33-8740-4009-8e8a-4c99647ac0b0-kube-api-access-94bmc\") pod \"opendatahub-operator-controller-manager-66d654f8fd-hsdn8\" (UID: \"358e1f33-8740-4009-8e8a-4c99647ac0b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.864510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.864476    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:13.933181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:13.933149    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:14.087040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:14.087003    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:14.389540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:14.389511    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"]
Oct 30 22:06:14.390800 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:14.390776    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358e1f33_8740_4009_8e8a_4c99647ac0b0.slice/crio-b08e3070e2fcdd61ca47f658d5d1dd057a07f710ad47a8c081a2f1a556aaa9a2 WatchSource:0}: Error finding container b08e3070e2fcdd61ca47f658d5d1dd057a07f710ad47a8c081a2f1a556aaa9a2: Status 404 returned error can't find the container with id b08e3070e2fcdd61ca47f658d5d1dd057a07f710ad47a8c081a2f1a556aaa9a2
Oct 30 22:06:14.392531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:14.392514    5028 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:14.846602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:14.846567    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8" event={"ID":"358e1f33-8740-4009-8e8a-4c99647ac0b0","Type":"ContainerStarted","Data":"b08e3070e2fcdd61ca47f658d5d1dd057a07f710ad47a8c081a2f1a556aaa9a2"}
Oct 30 22:06:16.053399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:16.053356    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:17.054615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:17.054582    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:17.268499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:17.268465    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:17.435106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:17.435024    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:18.867788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:18.867757    5028 generic.go:356] "Generic (PLEG): container finished" podID="358e1f33-8740-4009-8e8a-4c99647ac0b0" containerID="add8a18ed6fb16ad5e7d50cd4dead305ecb0b73a458b9b56c73ade3e72a3fe8a" exitCode=0
Oct 30 22:06:18.868168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:18.867807    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8" event={"ID":"358e1f33-8740-4009-8e8a-4c99647ac0b0","Type":"ContainerDied","Data":"add8a18ed6fb16ad5e7d50cd4dead305ecb0b73a458b9b56c73ade3e72a3fe8a"}
Oct 30 22:06:20.877800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:20.877760    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8" event={"ID":"358e1f33-8740-4009-8e8a-4c99647ac0b0","Type":"ContainerStarted","Data":"3a3476af0b6b037a36b40dea8e48ce1e51e23dae3cb0e9a1726c516e86f774ba"}
Oct 30 22:06:20.878188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:20.877852    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:21.177923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.177825    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8" podStartSLOduration=3.117806244 podStartE2EDuration="8.177810464s" totalImagesPullingTime="4.589820411s" totalInitContainerRuntime="470.183809ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:21.176159427 +0000 UTC m=+312.025672422" watchObservedRunningTime="2026-10-30 22:06:21.177810464 +0000 UTC m=+312.027323436"
Oct 30 22:06:21.615612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.615576    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:21.924489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.924409    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"]
Oct 30 22:06:21.927675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.927658    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:21.980019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.979992    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gg5w2\""
Oct 30 22:06:21.980562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.980547    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 22:06:21.980637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.980620    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 22:06:21.985818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:21.985788    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"]
Oct 30 22:06:22.039062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:22.039011    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdbf\" (UniqueName: \"kubernetes.io/projected/4c8e1dae-6c09-40ea-929a-17f1e515a6f2-kube-api-access-wwdbf\") pod \"authorino-operator-7b7bb4bc77-rcnrs\" (UID: \"4c8e1dae-6c09-40ea-929a-17f1e515a6f2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:22.139928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:22.139888    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdbf\" (UniqueName: \"kubernetes.io/projected/4c8e1dae-6c09-40ea-929a-17f1e515a6f2-kube-api-access-wwdbf\") pod \"authorino-operator-7b7bb4bc77-rcnrs\" (UID: \"4c8e1dae-6c09-40ea-929a-17f1e515a6f2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:22.198785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:22.198713    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdbf\" (UniqueName: \"kubernetes.io/projected/4c8e1dae-6c09-40ea-929a-17f1e515a6f2-kube-api-access-wwdbf\") pod \"authorino-operator-7b7bb4bc77-rcnrs\" (UID: \"4c8e1dae-6c09-40ea-929a-17f1e515a6f2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:22.240492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:22.240459    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:22.616717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:22.616678    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"]
Oct 30 22:06:22.618392 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:22.618362    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8e1dae_6c09_40ea_929a_17f1e515a6f2.slice/crio-5253b4c434401c033d20637fcef707442b0f9be63dbf146946a2359731365a6d WatchSource:0}: Error finding container 5253b4c434401c033d20637fcef707442b0f9be63dbf146946a2359731365a6d: Status 404 returned error can't find the container with id 5253b4c434401c033d20637fcef707442b0f9be63dbf146946a2359731365a6d
Oct 30 22:06:22.889521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:22.889432    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs" event={"ID":"4c8e1dae-6c09-40ea-929a-17f1e515a6f2","Type":"ContainerStarted","Data":"5253b4c434401c033d20637fcef707442b0f9be63dbf146946a2359731365a6d"}
Oct 30 22:06:23.238698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:23.238652    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:25.599561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.599524    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"]
Oct 30 22:06:25.603700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.603673    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.654992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.654957    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"]
Oct 30 22:06:25.673109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.673073    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8rt\" (UniqueName: \"kubernetes.io/projected/988ff033-85bb-4fe6-a517-52c372009cf6-kube-api-access-sg8rt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whn7d\" (UID: \"988ff033-85bb-4fe6-a517-52c372009cf6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.673285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.673240    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/988ff033-85bb-4fe6-a517-52c372009cf6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whn7d\" (UID: \"988ff033-85bb-4fe6-a517-52c372009cf6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.699970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.699936    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:25.700236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.700203    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nkx46\""
Oct 30 22:06:25.700357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.700256    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:25.774163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.774126    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8rt\" (UniqueName: \"kubernetes.io/projected/988ff033-85bb-4fe6-a517-52c372009cf6-kube-api-access-sg8rt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whn7d\" (UID: \"988ff033-85bb-4fe6-a517-52c372009cf6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.774342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.774205    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/988ff033-85bb-4fe6-a517-52c372009cf6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whn7d\" (UID: \"988ff033-85bb-4fe6-a517-52c372009cf6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.774682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.774610    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/988ff033-85bb-4fe6-a517-52c372009cf6-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whn7d\" (UID: \"988ff033-85bb-4fe6-a517-52c372009cf6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.833721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.833634    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8rt\" (UniqueName: \"kubernetes.io/projected/988ff033-85bb-4fe6-a517-52c372009cf6-kube-api-access-sg8rt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whn7d\" (UID: \"988ff033-85bb-4fe6-a517-52c372009cf6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:25.919455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:25.919368    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"
Oct 30 22:06:26.361005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.360974    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"]
Oct 30 22:06:26.364559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.364537    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:26.380173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.380148    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnlb\" (UniqueName: \"kubernetes.io/projected/779e849b-6f55-4500-9b81-aba988308aba-kube-api-access-4dnlb\") pod \"dns-operator-controller-manager-8557cbdb84-zs6bt\" (UID: \"779e849b-6f55-4500-9b81-aba988308aba\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:26.408319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.408288    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d"]
Oct 30 22:06:26.409256 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:26.409222    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988ff033_85bb_4fe6_a517_52c372009cf6.slice/crio-3be4ef3a378a0be4eee7e9383c1e6dbc7a54dd4d1a6013f4ab6c4315ebed7a74 WatchSource:0}: Error finding container 3be4ef3a378a0be4eee7e9383c1e6dbc7a54dd4d1a6013f4ab6c4315ebed7a74: Status 404 returned error can't find the container with id 3be4ef3a378a0be4eee7e9383c1e6dbc7a54dd4d1a6013f4ab6c4315ebed7a74
Oct 30 22:06:26.438207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.438179    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-btgng\""
Oct 30 22:06:26.438369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.438235    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 22:06:26.441091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.441064    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"]
Oct 30 22:06:26.479629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.479599    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:26.480749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.480719    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnlb\" (UniqueName: \"kubernetes.io/projected/779e849b-6f55-4500-9b81-aba988308aba-kube-api-access-4dnlb\") pod \"dns-operator-controller-manager-8557cbdb84-zs6bt\" (UID: \"779e849b-6f55-4500-9b81-aba988308aba\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:26.545383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.545351    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnlb\" (UniqueName: \"kubernetes.io/projected/779e849b-6f55-4500-9b81-aba988308aba-kube-api-access-4dnlb\") pod \"dns-operator-controller-manager-8557cbdb84-zs6bt\" (UID: \"779e849b-6f55-4500-9b81-aba988308aba\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:26.678363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.678287    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:26.856959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.856926    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:26.910834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:26.910800    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d" event={"ID":"988ff033-85bb-4fe6-a517-52c372009cf6","Type":"ContainerStarted","Data":"3be4ef3a378a0be4eee7e9383c1e6dbc7a54dd4d1a6013f4ab6c4315ebed7a74"}
Oct 30 22:06:27.004937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:27.004895    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:27.059860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:27.059827    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"]
Oct 30 22:06:27.061246 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:27.061221    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779e849b_6f55_4500_9b81_aba988308aba.slice/crio-d11df58a5dae22d67ae036b712f909a92cfc453d6784104c45d24a13676119e6 WatchSource:0}: Error finding container d11df58a5dae22d67ae036b712f909a92cfc453d6784104c45d24a13676119e6: Status 404 returned error can't find the container with id d11df58a5dae22d67ae036b712f909a92cfc453d6784104c45d24a13676119e6
Oct 30 22:06:27.650482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:27.650402    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:27.918492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:27.918448    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt" event={"ID":"779e849b-6f55-4500-9b81-aba988308aba","Type":"ContainerStarted","Data":"d11df58a5dae22d67ae036b712f909a92cfc453d6784104c45d24a13676119e6"}
Oct 30 22:06:27.920284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:27.920258    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs" event={"ID":"4c8e1dae-6c09-40ea-929a-17f1e515a6f2","Type":"ContainerStarted","Data":"78ea9c693ec01e55e45d669ca939a6df227a7fcd74c1573b38a2d557a814272b"}
Oct 30 22:06:27.920420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:27.920404    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:28.042289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:28.042239    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs" podStartSLOduration=2.247133661 podStartE2EDuration="7.042223014s" totalImagesPullingTime="4.795089353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:28.040314896 +0000 UTC m=+318.889827870" watchObservedRunningTime="2026-10-30 22:06:28.042223014 +0000 UTC m=+318.891735986"
Oct 30 22:06:28.438819 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:06:28.438778    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:06:29.277593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:29.277545    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:29.940579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:29.940540    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d" event={"ID":"988ff033-85bb-4fe6-a517-52c372009cf6","Type":"ContainerStarted","Data":"b6cb298431be284b3c7c84342335bf97f5a73ff0213c9ff342ce56249a680c6c"}
Oct 30 22:06:29.948231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:29.948201    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:30.151089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:30.151034    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whn7d" podStartSLOduration=2.466293872 podStartE2EDuration="5.151017953s" totalImagesPullingTime="2.684724081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:30.103001059 +0000 UTC m=+320.952514032" watchObservedRunningTime="2026-10-30 22:06:30.151017953 +0000 UTC m=+321.000530928"
Oct 30 22:06:30.946803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:30.946761    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt" event={"ID":"779e849b-6f55-4500-9b81-aba988308aba","Type":"ContainerStarted","Data":"f06ed219402249480c491863401902b2d3fdc8a4d2976ca59dbb4641b4c60f68"}
Oct 30 22:06:31.550764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.550709    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:31.806061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.805972    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt" podStartSLOduration=2.863524805 podStartE2EDuration="5.805955633s" totalImagesPullingTime="2.942430828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:31.062671589 +0000 UTC m=+321.912184562" watchObservedRunningTime="2026-10-30 22:06:31.805955633 +0000 UTC m=+322.655468605"
Oct 30 22:06:31.807282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.807257    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"]
Oct 30 22:06:31.810939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.810918    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:31.827734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.827711    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpl8\" (UniqueName: \"kubernetes.io/projected/274237e7-b77d-4d95-92ab-a3cb728bf913-kube-api-access-6rpl8\") pod \"limitador-operator-controller-manager-5778bb4449-xcz7l\" (UID: \"274237e7-b77d-4d95-92ab-a3cb728bf913\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:31.884886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.884857    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-hsdn8"
Oct 30 22:06:31.913374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.913345    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dc9qw\""
Oct 30 22:06:31.915430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.915402    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"]
Oct 30 22:06:31.916092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.916051    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:31.928304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.928273    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpl8\" (UniqueName: \"kubernetes.io/projected/274237e7-b77d-4d95-92ab-a3cb728bf913-kube-api-access-6rpl8\") pod \"limitador-operator-controller-manager-5778bb4449-xcz7l\" (UID: \"274237e7-b77d-4d95-92ab-a3cb728bf913\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:31.951221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.951195    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:31.991707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:31.991658    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpl8\" (UniqueName: \"kubernetes.io/projected/274237e7-b77d-4d95-92ab-a3cb728bf913-kube-api-access-6rpl8\") pod \"limitador-operator-controller-manager-5778bb4449-xcz7l\" (UID: \"274237e7-b77d-4d95-92ab-a3cb728bf913\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:32.084871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:32.084789    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:32.123854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:32.123819    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:32.492432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:32.492406    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"]
Oct 30 22:06:32.494049 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:32.494024    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274237e7_b77d_4d95_92ab_a3cb728bf913.slice/crio-7d99c578d428b7833a4755ef38836c082adf9cdef3392535dc08ef27dd73cdfa WatchSource:0}: Error finding container 7d99c578d428b7833a4755ef38836c082adf9cdef3392535dc08ef27dd73cdfa: Status 404 returned error can't find the container with id 7d99c578d428b7833a4755ef38836c082adf9cdef3392535dc08ef27dd73cdfa
Oct 30 22:06:32.508266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:32.508233    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:32.694035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:32.693997    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:32.957978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:32.957952    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l" event={"ID":"274237e7-b77d-4d95-92ab-a3cb728bf913","Type":"ContainerStarted","Data":"7d99c578d428b7833a4755ef38836c082adf9cdef3392535dc08ef27dd73cdfa"}
Oct 30 22:06:33.046082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:33.046044    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:33.736850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:33.736816    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:33.982622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:33.982584    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:34.257352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:34.257318    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:34.823697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:34.823648    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:36.979537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:36.979501    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l" event={"ID":"274237e7-b77d-4d95-92ab-a3cb728bf913","Type":"ContainerStarted","Data":"c19c057f0315dd225b5130a4f3a0952b8083d235523f68837c0a6302afcc3249"}
Oct 30 22:06:36.980047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:36.979710    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:37.095607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.095557    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l" podStartSLOduration=2.218948576 podStartE2EDuration="6.095543337s" totalImagesPullingTime="3.876594761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:37.094288612 +0000 UTC m=+327.943801586" watchObservedRunningTime="2026-10-30 22:06:37.095543337 +0000 UTC m=+327.945056310"
Oct 30 22:06:37.895424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.895381    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"]
Oct 30 22:06:37.897955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.897933    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:37.952101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.952070    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"]
Oct 30 22:06:37.976451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.976422    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-99dlf\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:37.976597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.976455    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9vp\" (UniqueName: \"kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp\") pod \"kuadrant-operator-controller-manager-678d87766f-99dlf\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:37.997015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:37.996988    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nhrgr\""
Oct 30 22:06:38.077192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.077158    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-99dlf\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:38.077192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.077191    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9vp\" (UniqueName: \"kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp\") pod \"kuadrant-operator-controller-manager-678d87766f-99dlf\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:38.077621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.077600    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-99dlf\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:38.138278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.138249    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9vp\" (UniqueName: \"kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp\") pod \"kuadrant-operator-controller-manager-678d87766f-99dlf\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:38.208970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.208946    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:38.666940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.666915    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"]
Oct 30 22:06:38.668397 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:38.668373    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6194d8_8f7b_46aa_a3dc_c98949a2fba9.slice/crio-11f09bea5d4bf5669f25d9bbfed985126e226cb3cafddb99c86d9b12f6ca2861 WatchSource:0}: Error finding container 11f09bea5d4bf5669f25d9bbfed985126e226cb3cafddb99c86d9b12f6ca2861: Status 404 returned error can't find the container with id 11f09bea5d4bf5669f25d9bbfed985126e226cb3cafddb99c86d9b12f6ca2861
Oct 30 22:06:38.930071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.929994    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rcnrs"
Oct 30 22:06:38.990498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:38.990462    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf" event={"ID":"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9","Type":"ContainerStarted","Data":"11f09bea5d4bf5669f25d9bbfed985126e226cb3cafddb99c86d9b12f6ca2861"}
Oct 30 22:06:41.307236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:41.307198    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:41.452177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:41.452136    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:41.649020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:41.648933    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:41.795539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:41.795501    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:42.300421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:42.300381    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:42.498420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:42.498388    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:42.672953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:42.672868    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:42.920622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:42.920573    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:42.960895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:42.960869    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zs6bt"
Oct 30 22:06:43.143497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:43.143464    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:43.323323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:43.323236    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.017799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.017710    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf" event={"ID":"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9","Type":"ContainerStarted","Data":"e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca"}
Oct 30 22:06:44.018241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.017817    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:44.230977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.230913    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf" podStartSLOduration=2.156677603 podStartE2EDuration="7.230893082s" totalImagesPullingTime="5.074215479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:44.230776735 +0000 UTC m=+335.080289712" watchObservedRunningTime="2026-10-30 22:06:44.230893082 +0000 UTC m=+335.080406057"
Oct 30 22:06:44.287593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.287510    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.484049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.484005    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.769890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.769852    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"]
Oct 30 22:06:44.772450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.772433    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.826573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.826534    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wb45w\""
Oct 30 22:06:44.833584 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.833538    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"]
Oct 30 22:06:44.834519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834493    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834534    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834607    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834706    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834773    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834801    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7nf\" (UniqueName: \"kubernetes.io/projected/d409403f-fbdb-4f41-bc4f-9f41afc66d88-kube-api-access-dz7nf\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834868    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834911    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.834999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.834960    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.866613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.866583    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.936297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936251    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936289    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7nf\" (UniqueName: \"kubernetes.io/projected/d409403f-fbdb-4f41-bc4f-9f41afc66d88-kube-api-access-dz7nf\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936324    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936342    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936361    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936660    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936752    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936785    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936789    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.936895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.936815    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.937080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.937057    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.937141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.937114    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.937199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.937170    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.937199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.937174    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.938613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.938584    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:44.938959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:44.938939    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:45.002988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.002954    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d409403f-fbdb-4f41-bc4f-9f41afc66d88-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:45.002988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.002987    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7nf\" (UniqueName: \"kubernetes.io/projected/d409403f-fbdb-4f41-bc4f-9f41afc66d88-kube-api-access-dz7nf\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-th9dm\" (UID: \"d409403f-fbdb-4f41-bc4f-9f41afc66d88\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:45.027230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.027144    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:45.084818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.084783    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:45.173567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.173508    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:45.500604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.500570    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"]
Oct 30 22:06:45.505522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.505485    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:45.505619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.505553    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:45.505619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:45.505580    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:06:46.027737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:46.027678    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm" event={"ID":"d409403f-fbdb-4f41-bc4f-9f41afc66d88","Type":"ContainerStarted","Data":"d9f63e428bcee9e28bef6d90a398592182b4382f74b2e4bff460bea1c469d563"}
Oct 30 22:06:46.028217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:46.027759    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm" event={"ID":"d409403f-fbdb-4f41-bc4f-9f41afc66d88","Type":"ContainerStarted","Data":"d895787aa94ea1c9ac0789ad3c1731734b7eac5ad6c85b1aa1979f6d91e82302"}
Oct 30 22:06:46.085532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:46.085495    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:46.090479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:46.090412    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:46.144702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:46.144636    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm" podStartSLOduration=2.144618166 podStartE2EDuration="2.144618166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:46.142171974 +0000 UTC m=+336.991684958" watchObservedRunningTime="2026-10-30 22:06:46.144618166 +0000 UTC m=+336.994131218"
Oct 30 22:06:47.033464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:47.033429    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:47.034355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:47.034336    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-th9dm"
Oct 30 22:06:47.450415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:47.450333    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:47.600626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:47.600592    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:47.986969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:47.986937    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xcz7l"
Oct 30 22:06:48.088147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:48.088104    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:48.465601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:48.465562    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:48.637177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:48.637113    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:48.786706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:48.786601    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:49.720614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.720577    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lnmbf"]
Oct 30 22:06:49.726410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.726384    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:49.776712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.776663    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lnmbf"]
Oct 30 22:06:49.779613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.779586    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpqsz\" (UniqueName: \"kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz\") pod \"authorino-57f7f8659f-lnmbf\" (UID: \"2e35f080-9af2-4ba2-a639-ae09749eca9e\") " pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:49.821886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.821860    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ll7qr\""
Oct 30 22:06:49.873390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.873353    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:49.880547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.880522    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpqsz\" (UniqueName: \"kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz\") pod \"authorino-57f7f8659f-lnmbf\" (UID: \"2e35f080-9af2-4ba2-a639-ae09749eca9e\") " pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:49.939889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:49.939860    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpqsz\" (UniqueName: \"kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz\") pod \"authorino-57f7f8659f-lnmbf\" (UID: \"2e35f080-9af2-4ba2-a639-ae09749eca9e\") " pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:50.018542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:50.018457    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.050748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:50.050712    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:50.411497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:50.411467    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lnmbf"]
Oct 30 22:06:50.412748 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:50.412721    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e35f080_9af2_4ba2_a639_ae09749eca9e.slice/crio-d9ef85257d542d2e47a405b9b88f2a513938e2b4f513fe1f851b01c184ce9add WatchSource:0}: Error finding container d9ef85257d542d2e47a405b9b88f2a513938e2b4f513fe1f851b01c184ce9add: Status 404 returned error can't find the container with id d9ef85257d542d2e47a405b9b88f2a513938e2b4f513fe1f851b01c184ce9add
Oct 30 22:06:50.837919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:50.837881    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.986544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:50.986509    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:51.054310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:51.054267    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lnmbf" event={"ID":"2e35f080-9af2-4ba2-a639-ae09749eca9e","Type":"ContainerStarted","Data":"d9ef85257d542d2e47a405b9b88f2a513938e2b4f513fe1f851b01c184ce9add"}
Oct 30 22:06:51.597371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:51.597334    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:52.724712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.724657    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ghss5"]
Oct 30 22:06:52.727667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.727639    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:06:52.782156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.782123    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ghss5"]
Oct 30 22:06:52.808538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.808502    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nj4d\" (UniqueName: \"kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d\") pod \"authorino-56c6c9ddd5-ghss5\" (UID: \"3394b0f1-b32f-4548-8922-23292369e1cf\") " pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:06:52.866066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.866032    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:52.894665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.894635    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"]
Oct 30 22:06:52.897644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.897615    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:52.909564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.909534    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nj4d\" (UniqueName: \"kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d\") pod \"authorino-56c6c9ddd5-ghss5\" (UID: \"3394b0f1-b32f-4548-8922-23292369e1cf\") " pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:06:52.949830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.949586    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 22:06:52.949830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.949621    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bcwpx\""
Oct 30 22:06:52.949830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.949783    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 22:06:52.952377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.952349    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"]
Oct 30 22:06:52.968354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:52.968325    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nj4d\" (UniqueName: \"kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d\") pod \"authorino-56c6c9ddd5-ghss5\" (UID: \"3394b0f1-b32f-4548-8922-23292369e1cf\") " pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:06:53.010767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.010652    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a17310a7-9513-449e-a500-ffb813321d7b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.010940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.010810    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krngt\" (UniqueName: \"kubernetes.io/projected/a17310a7-9513-449e-a500-ffb813321d7b-kube-api-access-krngt\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.010940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.010864    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a17310a7-9513-449e-a500-ffb813321d7b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.041494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.041459    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:06:53.099828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.099682    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:53.113414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.113141    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krngt\" (UniqueName: \"kubernetes.io/projected/a17310a7-9513-449e-a500-ffb813321d7b-kube-api-access-krngt\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.113414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.113217    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a17310a7-9513-449e-a500-ffb813321d7b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.113414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.113270    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a17310a7-9513-449e-a500-ffb813321d7b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.113944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.113917    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a17310a7-9513-449e-a500-ffb813321d7b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.123076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.123053    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a17310a7-9513-449e-a500-ffb813321d7b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.172966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.172928    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krngt\" (UniqueName: \"kubernetes.io/projected/a17310a7-9513-449e-a500-ffb813321d7b-kube-api-access-krngt\") pod \"kuadrant-console-plugin-66c4fb575d-lbp5n\" (UID: \"a17310a7-9513-449e-a500-ffb813321d7b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.216629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.216591    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"
Oct 30 22:06:53.412184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.412152    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ghss5"]
Oct 30 22:06:53.432299 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:53.432265    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3394b0f1_b32f_4548_8922_23292369e1cf.slice/crio-0662a28e73d5b97627473c28cad92ddf5c130994e978baa9aa8632d6cb12386a WatchSource:0}: Error finding container 0662a28e73d5b97627473c28cad92ddf5c130994e978baa9aa8632d6cb12386a: Status 404 returned error can't find the container with id 0662a28e73d5b97627473c28cad92ddf5c130994e978baa9aa8632d6cb12386a
Oct 30 22:06:53.584260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:53.584234    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n"]
Oct 30 22:06:53.585453 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:06:53.585426    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda17310a7_9513_449e_a500_ffb813321d7b.slice/crio-ad5b1911b7886bbbcbada0f01777c79b0db6b65b07026dc70fb57187ea93c3b2 WatchSource:0}: Error finding container ad5b1911b7886bbbcbada0f01777c79b0db6b65b07026dc70fb57187ea93c3b2: Status 404 returned error can't find the container with id ad5b1911b7886bbbcbada0f01777c79b0db6b65b07026dc70fb57187ea93c3b2
Oct 30 22:06:54.071603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:54.071565    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n" event={"ID":"a17310a7-9513-449e-a500-ffb813321d7b","Type":"ContainerStarted","Data":"ad5b1911b7886bbbcbada0f01777c79b0db6b65b07026dc70fb57187ea93c3b2"}
Oct 30 22:06:54.072746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:54.072713    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5" event={"ID":"3394b0f1-b32f-4548-8922-23292369e1cf","Type":"ContainerStarted","Data":"0662a28e73d5b97627473c28cad92ddf5c130994e978baa9aa8632d6cb12386a"}
Oct 30 22:06:54.074113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:54.074090    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lnmbf" event={"ID":"2e35f080-9af2-4ba2-a639-ae09749eca9e","Type":"ContainerStarted","Data":"114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f"}
Oct 30 22:06:54.182822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:54.182770    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lnmbf" podStartSLOduration=2.157820529 podStartE2EDuration="5.182756356s" totalImagesPullingTime="3.024935827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:54.181998765 +0000 UTC m=+345.031511739" watchObservedRunningTime="2026-10-30 22:06:54.182756356 +0000 UTC m=+345.032269329"
Oct 30 22:06:55.024557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:55.024532    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 22:06:55.080537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:55.080503    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5" event={"ID":"3394b0f1-b32f-4548-8922-23292369e1cf","Type":"ContainerStarted","Data":"cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985"}
Oct 30 22:06:55.297105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:55.297007    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5" podStartSLOduration=1.746691239 podStartE2EDuration="3.29699254s" totalImagesPullingTime="1.550301301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:55.295144259 +0000 UTC m=+346.144657233" watchObservedRunningTime="2026-10-30 22:06:55.29699254 +0000 UTC m=+346.146505513"
Oct 30 22:06:55.323188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:55.323149    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lnmbf"]
Oct 30 22:06:55.660649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:55.660537    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:55.850915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:55.850778    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.088000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:56.087919    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lnmbf" podUID="2e35f080-9af2-4ba2-a639-ae09749eca9e" containerName="authorino" containerID="cri-o://114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f" gracePeriod=30
Oct 30 22:06:56.359940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:56.359916    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:56.441064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:56.441024    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz\" (UniqueName: \"kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz\") pod \"2e35f080-9af2-4ba2-a639-ae09749eca9e\" (UID: \"2e35f080-9af2-4ba2-a639-ae09749eca9e\") "
Oct 30 22:06:56.443214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:56.443192    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz" pod "2e35f080-9af2-4ba2-a639-ae09749eca9e" (UID: "2e35f080-9af2-4ba2-a639-ae09749eca9e"). InnerVolumeSpecName "kube-api-access-lpqsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:56.542159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:56.542118    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpqsz\" (UniqueName: \"kubernetes.io/projected/2e35f080-9af2-4ba2-a639-ae09749eca9e-kube-api-access-lpqsz\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:57.093886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.093851    5028 generic.go:356] "Generic (PLEG): container finished" podID="2e35f080-9af2-4ba2-a639-ae09749eca9e" containerID="114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f" exitCode=0
Oct 30 22:06:57.094369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.093912    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lnmbf"
Oct 30 22:06:57.094369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.093942    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lnmbf" event={"ID":"2e35f080-9af2-4ba2-a639-ae09749eca9e","Type":"ContainerDied","Data":"114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f"}
Oct 30 22:06:57.094369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.093988    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lnmbf" event={"ID":"2e35f080-9af2-4ba2-a639-ae09749eca9e","Type":"ContainerDied","Data":"d9ef85257d542d2e47a405b9b88f2a513938e2b4f513fe1f851b01c184ce9add"}
Oct 30 22:06:57.094369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.094008    5028 scope.go:122] "RemoveContainer" containerID="114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f"
Oct 30 22:06:57.105110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.105087    5028 scope.go:122] "RemoveContainer" containerID="114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f"
Oct 30 22:06:57.105401 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:06:57.105373    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f\": container with ID starting with 114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f not found: ID does not exist" containerID="114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f"
Oct 30 22:06:57.105502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.105415    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f"} err="failed to get container status \"114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f\": rpc error: code = NotFound desc = could not find container \"114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f\": container with ID starting with 114939dc5dfbcce4c48e7232ebab1578db173951b110e362ef62e8108cbd2a0f not found: ID does not exist"
Oct 30 22:06:57.253908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.253873    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lnmbf"]
Oct 30 22:06:57.258048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.258017    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lnmbf"]
Oct 30 22:06:57.292409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:06:57.292382    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e35f080-9af2-4ba2-a639-ae09749eca9e" path="/var/lib/kubelet/pods/2e35f080-9af2-4ba2-a639-ae09749eca9e/volumes"
Oct 30 22:06:58.438569 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:06:58.438525    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:07:00.115981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:00.115945    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n" event={"ID":"a17310a7-9513-449e-a500-ffb813321d7b","Type":"ContainerStarted","Data":"26f19a55acf50781a3a92f9b7b81735a447832899ff4aabb052fbe7ba15c8d70"}
Oct 30 22:07:00.226303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:00.226252    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lbp5n" podStartSLOduration=1.921352933 podStartE2EDuration="8.22623825s" totalImagesPullingTime="6.304885317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:00.225816454 +0000 UTC m=+351.075329425" watchObservedRunningTime="2026-10-30 22:07:00.22623825 +0000 UTC m=+351.075751223"
Oct 30 22:07:01.075638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:01.075601    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:07.576013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:07.575977    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:07.727658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:07.727621    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:07.872523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:07.872436    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:09.173958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.173925    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"]
Oct 30 22:07:09.174446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.174325    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2e35f080-9af2-4ba2-a639-ae09749eca9e" containerName="authorino"
Oct 30 22:07:09.174446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.174337    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e35f080-9af2-4ba2-a639-ae09749eca9e" containerName="authorino"
Oct 30 22:07:09.174446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.174413    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e35f080-9af2-4ba2-a639-ae09749eca9e" containerName="authorino"
Oct 30 22:07:09.177591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.177575    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.251053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.250978    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed489f43-65e0-4f32-ac7f-1e1e16c12572-cert\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.251238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.251128    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed489f43-65e0-4f32-ac7f-1e1e16c12572-metrics-certs\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.251238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.251171    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed489f43-65e0-4f32-ac7f-1e1e16c12572-manager-config\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.251328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.251256    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvns\" (UniqueName: \"kubernetes.io/projected/ed489f43-65e0-4f32-ac7f-1e1e16c12572-kube-api-access-5jvns\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.277671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.277635    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 22:07:09.277874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.277737    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-chwx4\""
Oct 30 22:07:09.277874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.277786    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 22:07:09.277874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.277806    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:07:09.277874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.277843    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 22:07:09.278097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.277912    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 22:07:09.279999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.279979    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"]
Oct 30 22:07:09.352015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.351983    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed489f43-65e0-4f32-ac7f-1e1e16c12572-cert\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.352172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.352108    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed489f43-65e0-4f32-ac7f-1e1e16c12572-metrics-certs\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.352172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.352161    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed489f43-65e0-4f32-ac7f-1e1e16c12572-manager-config\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.352394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.352204    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvns\" (UniqueName: \"kubernetes.io/projected/ed489f43-65e0-4f32-ac7f-1e1e16c12572-kube-api-access-5jvns\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.352849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.352828    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed489f43-65e0-4f32-ac7f-1e1e16c12572-manager-config\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.354468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.354449    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed489f43-65e0-4f32-ac7f-1e1e16c12572-cert\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.354547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.354500    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed489f43-65e0-4f32-ac7f-1e1e16c12572-metrics-certs\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.413633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.413603    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvns\" (UniqueName: \"kubernetes.io/projected/ed489f43-65e0-4f32-ac7f-1e1e16c12572-kube-api-access-5jvns\") pod \"jobset-controller-manager-5dbc98fd88-hg4db\" (UID: \"ed489f43-65e0-4f32-ac7f-1e1e16c12572\") " pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.488485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.488452    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:09.861368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:09.861343    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"]
Oct 30 22:07:09.863056 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:07:09.863026    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded489f43_65e0_4f32_ac7f_1e1e16c12572.slice/crio-db848cdfec795549eaf196b7198a9d35691d92761efac7a19a838539acd4c47c WatchSource:0}: Error finding container db848cdfec795549eaf196b7198a9d35691d92761efac7a19a838539acd4c47c: Status 404 returned error can't find the container with id db848cdfec795549eaf196b7198a9d35691d92761efac7a19a838539acd4c47c
Oct 30 22:07:10.168730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:10.168623    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db" event={"ID":"ed489f43-65e0-4f32-ac7f-1e1e16c12572","Type":"ContainerStarted","Data":"db848cdfec795549eaf196b7198a9d35691d92761efac7a19a838539acd4c47c"}
Oct 30 22:07:14.191395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:14.191303    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db" event={"ID":"ed489f43-65e0-4f32-ac7f-1e1e16c12572","Type":"ContainerStarted","Data":"df825b69aa994970e7b776cd42519e499b476d07d86361a9af36f5f52ea0c9d1"}
Oct 30 22:07:14.191851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:14.191416    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:14.307726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:14.307652    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db" podStartSLOduration=1.38963466 podStartE2EDuration="5.307639613s" totalImagesPullingTime="3.918004953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:14.306407798 +0000 UTC m=+365.155920781" watchObservedRunningTime="2026-10-30 22:07:14.307639613 +0000 UTC m=+365.157152585"
Oct 30 22:07:15.254160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:15.254124    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:15.420959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:15.420926    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:15.954381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:15.954351    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:16.121923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:16.121892    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:23.941774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:23.941734    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:25.202946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:25.202912    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5dbc98fd88-hg4db"
Oct 30 22:07:28.434144 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:07:28.434108    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:07:37.216386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:37.216341    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:37.374334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:37.374298    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:37.521089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:37.521010    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:37.669678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:37.669643    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:41.895493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:41.895458    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:42.055034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:42.054993    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:44.844320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:44.844287    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:45.075622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:07:45.075583    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:58.435035 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:07:58.434961    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:08:01.456550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:01.456508    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:07.540078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:07.540037    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:07.712085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:07.712044    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:10.021762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:10.021717    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:10.180749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:10.180715    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:12.074112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:12.074056    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:12.226487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:12.226441    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:12.397839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:12.397751    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:12.562330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:12.562297    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:15.779998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:15.779954    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:15.931999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:15.931967    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:16.167318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:16.167227    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:16.312177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:16.312141    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:16.489043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:16.489006    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:16.639945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:16.639910    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:16.793822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:16.793740    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:16.943878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:16.943839    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.109789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:17.109683    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.263250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:17.263213    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:18.316917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:18.316886    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.165964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:23.165928    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.331775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:23.331738    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.474843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:23.474806    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.623033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:23.623001    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.778736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:23.778641    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.981899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:23.981867    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:24.133493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.133409    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:24.343423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.343385    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:24.489849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.489813    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w857q"]
Oct 30 22:08:24.493331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.493313    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:24.497526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.497504    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lv7\" (UniqueName: \"kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7\") pod \"authorino-6f6944bcdd-w857q\" (UID: \"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d\") " pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:24.499816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.499786    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:24.549960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.549921    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w857q"]
Oct 30 22:08:24.598563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.598524    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lv7\" (UniqueName: \"kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7\") pod \"authorino-6f6944bcdd-w857q\" (UID: \"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d\") " pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:24.659395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.659357    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lv7\" (UniqueName: \"kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7\") pod \"authorino-6f6944bcdd-w857q\" (UID: \"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d\") " pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:24.804586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.804497    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:24.867051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.867012    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w857q"]
Oct 30 22:08:24.893241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.893207    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c6df5c77-jjjfr"]
Oct 30 22:08:24.897005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.896981    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:24.900545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.900519    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwrv\" (UniqueName: \"kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv\") pod \"authorino-7c6df5c77-jjjfr\" (UID: \"07787cb1-6a30-43a4-9a9f-270e03117bac\") " pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:24.949527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:24.949497    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6df5c77-jjjfr"]
Oct 30 22:08:25.001341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.001310    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwrv\" (UniqueName: \"kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv\") pod \"authorino-7c6df5c77-jjjfr\" (UID: \"07787cb1-6a30-43a4-9a9f-270e03117bac\") " pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:25.060479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.060411    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwrv\" (UniqueName: \"kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv\") pod \"authorino-7c6df5c77-jjjfr\" (UID: \"07787cb1-6a30-43a4-9a9f-270e03117bac\") " pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:25.174136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.174103    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w857q"]
Oct 30 22:08:25.179466 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:08:25.179440    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ae5fd05_1cb6_4b24_bc64_f2cbaf11918d.slice/crio-8a07c97dd77d86ba6b029922b0788fd5564042894f9037aaf72f1f9f76fe9206 WatchSource:0}: Error finding container 8a07c97dd77d86ba6b029922b0788fd5564042894f9037aaf72f1f9f76fe9206: Status 404 returned error can't find the container with id 8a07c97dd77d86ba6b029922b0788fd5564042894f9037aaf72f1f9f76fe9206
Oct 30 22:08:25.209409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.209381    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:25.324229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.324142    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6df5c77-jjjfr"]
Oct 30 22:08:25.352970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.352937    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6ff796694c-s4bfp"]
Oct 30 22:08:25.357526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.357504    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.405265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.405228    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87gr\" (UniqueName: \"kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr\") pod \"authorino-6ff796694c-s4bfp\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") " pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.405265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.405266    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert\") pod \"authorino-6ff796694c-s4bfp\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") " pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.408347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.408322    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 22:08:25.410553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.410526    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ff796694c-s4bfp"]
Oct 30 22:08:25.506159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.506117    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87gr\" (UniqueName: \"kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr\") pod \"authorino-6ff796694c-s4bfp\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") " pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.506159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.506161    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert\") pod \"authorino-6ff796694c-s4bfp\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") " pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.508564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.508543    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert\") pod \"authorino-6ff796694c-s4bfp\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") " pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.554193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.554155    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w857q" event={"ID":"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d","Type":"ContainerStarted","Data":"8a07c97dd77d86ba6b029922b0788fd5564042894f9037aaf72f1f9f76fe9206"}
Oct 30 22:08:25.565354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.565328    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87gr\" (UniqueName: \"kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr\") pod \"authorino-6ff796694c-s4bfp\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") " pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:25.576269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.576201    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6df5c77-jjjfr"]
Oct 30 22:08:25.582593 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:08:25.582566    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07787cb1_6a30_43a4_9a9f_270e03117bac.slice/crio-66469777075e4f7eb14af4e4c45be496a97b17ba59120ac2f725beb0c5f51f04 WatchSource:0}: Error finding container 66469777075e4f7eb14af4e4c45be496a97b17ba59120ac2f725beb0c5f51f04: Status 404 returned error can't find the container with id 66469777075e4f7eb14af4e4c45be496a97b17ba59120ac2f725beb0c5f51f04
Oct 30 22:08:25.670387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:25.670353    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:08:26.041316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:26.041284    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ff796694c-s4bfp"]
Oct 30 22:08:26.047358 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:08:26.047320    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be15d82_f54d_4237_8508_2b09ca5c4d8a.slice/crio-20c43bddd0374ca62c5c3c5d6340c43ac42f22712c486c5d966d109a0f77cfd0 WatchSource:0}: Error finding container 20c43bddd0374ca62c5c3c5d6340c43ac42f22712c486c5d966d109a0f77cfd0: Status 404 returned error can't find the container with id 20c43bddd0374ca62c5c3c5d6340c43ac42f22712c486c5d966d109a0f77cfd0
Oct 30 22:08:26.560331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:26.560292    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ff796694c-s4bfp" event={"ID":"6be15d82-f54d-4237-8508-2b09ca5c4d8a","Type":"ContainerStarted","Data":"20c43bddd0374ca62c5c3c5d6340c43ac42f22712c486c5d966d109a0f77cfd0"}
Oct 30 22:08:26.561423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:26.561394    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6df5c77-jjjfr" event={"ID":"07787cb1-6a30-43a4-9a9f-270e03117bac","Type":"ContainerStarted","Data":"66469777075e4f7eb14af4e4c45be496a97b17ba59120ac2f725beb0c5f51f04"}
Oct 30 22:08:27.568761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.568722    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w857q" event={"ID":"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d","Type":"ContainerStarted","Data":"7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a"}
Oct 30 22:08:27.569251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.568788    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-w857q" podUID="7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" containerName="authorino" containerID="cri-o://7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a" gracePeriod=30
Oct 30 22:08:27.570531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.570500    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ff796694c-s4bfp" event={"ID":"6be15d82-f54d-4237-8508-2b09ca5c4d8a","Type":"ContainerStarted","Data":"de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379"}
Oct 30 22:08:27.572790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.572760    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6df5c77-jjjfr" event={"ID":"07787cb1-6a30-43a4-9a9f-270e03117bac","Type":"ContainerStarted","Data":"ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1"}
Oct 30 22:08:27.572919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.572894    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7c6df5c77-jjjfr" podUID="07787cb1-6a30-43a4-9a9f-270e03117bac" containerName="authorino" containerID="cri-o://ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1" gracePeriod=30
Oct 30 22:08:27.680851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.680802    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-w857q" podStartSLOduration=1.786385788 podStartE2EDuration="3.680786897s" totalImagesPullingTime="1.894401109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:27.679327003 +0000 UTC m=+438.528839977" watchObservedRunningTime="2026-10-30 22:08:27.680786897 +0000 UTC m=+438.530299869"
Oct 30 22:08:27.790293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.790242    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6ff796694c-s4bfp" podStartSLOduration=1.500195919 podStartE2EDuration="2.790223554s" totalImagesPullingTime="1.290027635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:27.789645035 +0000 UTC m=+438.639158030" watchObservedRunningTime="2026-10-30 22:08:27.790223554 +0000 UTC m=+438.639736527"
Oct 30 22:08:27.821364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.821326    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ghss5"]
Oct 30 22:08:27.821586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.821553    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5" podUID="3394b0f1-b32f-4548-8922-23292369e1cf" containerName="authorino" containerID="cri-o://cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985" gracePeriod=30
Oct 30 22:08:27.899152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.899056    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c6df5c77-jjjfr" podStartSLOduration=2.356233426 podStartE2EDuration="3.899043634s" totalImagesPullingTime="1.542810208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:27.898463807 +0000 UTC m=+438.747976780" watchObservedRunningTime="2026-10-30 22:08:27.899043634 +0000 UTC m=+438.748556658"
Oct 30 22:08:27.937529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.937116    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:27.940881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:27.940853    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:28.024506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.024473    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7\" (UniqueName: \"kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7\") pod \"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d\" (UID: \"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d\") "
Oct 30 22:08:28.024712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.024527    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv\" (UniqueName: \"kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv\") pod \"07787cb1-6a30-43a4-9a9f-270e03117bac\" (UID: \"07787cb1-6a30-43a4-9a9f-270e03117bac\") "
Oct 30 22:08:28.026724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.026660    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv" pod "07787cb1-6a30-43a4-9a9f-270e03117bac" (UID: "07787cb1-6a30-43a4-9a9f-270e03117bac"). InnerVolumeSpecName "kube-api-access-xhwrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:28.028404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.028374    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7" pod "7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" (UID: "7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d"). InnerVolumeSpecName "kube-api-access-57lv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:28.058745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.058717    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:08:28.124970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.124935    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d\" (UniqueName: \"kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d\") pod \"3394b0f1-b32f-4548-8922-23292369e1cf\" (UID: \"3394b0f1-b32f-4548-8922-23292369e1cf\") "
Oct 30 22:08:28.125180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.125165    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lv7\" (UniqueName: \"kubernetes.io/projected/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d-kube-api-access-57lv7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:28.125229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.125183    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwrv\" (UniqueName: \"kubernetes.io/projected/07787cb1-6a30-43a4-9a9f-270e03117bac-kube-api-access-xhwrv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:28.127057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.127024    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d" pod "3394b0f1-b32f-4548-8922-23292369e1cf" (UID: "3394b0f1-b32f-4548-8922-23292369e1cf"). InnerVolumeSpecName "kube-api-access-2nj4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:28.226033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.225999    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nj4d\" (UniqueName: \"kubernetes.io/projected/3394b0f1-b32f-4548-8922-23292369e1cf-kube-api-access-2nj4d\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:28.441229 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:08:28.441189    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:08:28.579012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.578935    5028 generic.go:356] "Generic (PLEG): container finished" podID="3394b0f1-b32f-4548-8922-23292369e1cf" containerID="cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985" exitCode=0
Oct 30 22:08:28.579012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.578993    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5"
Oct 30 22:08:28.579498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.579015    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5" event={"ID":"3394b0f1-b32f-4548-8922-23292369e1cf","Type":"ContainerDied","Data":"cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985"}
Oct 30 22:08:28.579498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.579063    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ghss5" event={"ID":"3394b0f1-b32f-4548-8922-23292369e1cf","Type":"ContainerDied","Data":"0662a28e73d5b97627473c28cad92ddf5c130994e978baa9aa8632d6cb12386a"}
Oct 30 22:08:28.579498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.579086    5028 scope.go:122] "RemoveContainer" containerID="cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985"
Oct 30 22:08:28.580508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.580323    5028 generic.go:356] "Generic (PLEG): container finished" podID="7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" containerID="7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a" exitCode=0
Oct 30 22:08:28.580508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.580366    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w857q"
Oct 30 22:08:28.580508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.580404    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w857q" event={"ID":"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d","Type":"ContainerDied","Data":"7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a"}
Oct 30 22:08:28.580508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.580445    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-w857q" event={"ID":"7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d","Type":"ContainerDied","Data":"8a07c97dd77d86ba6b029922b0788fd5564042894f9037aaf72f1f9f76fe9206"}
Oct 30 22:08:28.581965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.581940    5028 generic.go:356] "Generic (PLEG): container finished" podID="07787cb1-6a30-43a4-9a9f-270e03117bac" containerID="ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1" exitCode=0
Oct 30 22:08:28.582068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.581975    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6df5c77-jjjfr"
Oct 30 22:08:28.582068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.582015    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6df5c77-jjjfr" event={"ID":"07787cb1-6a30-43a4-9a9f-270e03117bac","Type":"ContainerDied","Data":"ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1"}
Oct 30 22:08:28.582068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.582059    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6df5c77-jjjfr" event={"ID":"07787cb1-6a30-43a4-9a9f-270e03117bac","Type":"ContainerDied","Data":"66469777075e4f7eb14af4e4c45be496a97b17ba59120ac2f725beb0c5f51f04"}
Oct 30 22:08:28.588926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.588910    5028 scope.go:122] "RemoveContainer" containerID="cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985"
Oct 30 22:08:28.589220 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:08:28.589195    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985\": container with ID starting with cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985 not found: ID does not exist" containerID="cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985"
Oct 30 22:08:28.589284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.589227    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985"} err="failed to get container status \"cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985\": rpc error: code = NotFound desc = could not find container \"cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985\": container with ID starting with cb43c17dfc77838a1438e318599b50f24835b6d8fdecdcc33a8c37be5ed88985 not found: ID does not exist"
Oct 30 22:08:28.589284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.589252    5028 scope.go:122] "RemoveContainer" containerID="7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a"
Oct 30 22:08:28.600150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.600127    5028 scope.go:122] "RemoveContainer" containerID="7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a"
Oct 30 22:08:28.600783 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:08:28.600753    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a\": container with ID starting with 7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a not found: ID does not exist" containerID="7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a"
Oct 30 22:08:28.600855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.600792    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a"} err="failed to get container status \"7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a\": rpc error: code = NotFound desc = could not find container \"7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a\": container with ID starting with 7c8a5e29cdd525d7939f150285d89eaaa48827d5704ab5eeeb8abb86531afe2a not found: ID does not exist"
Oct 30 22:08:28.600855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.600815    5028 scope.go:122] "RemoveContainer" containerID="ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1"
Oct 30 22:08:28.609108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.609088    5028 scope.go:122] "RemoveContainer" containerID="ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1"
Oct 30 22:08:28.609364 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:08:28.609343    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1\": container with ID starting with ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1 not found: ID does not exist" containerID="ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1"
Oct 30 22:08:28.609419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.609375    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1"} err="failed to get container status \"ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1\": rpc error: code = NotFound desc = could not find container \"ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1\": container with ID starting with ee2697f2ccd00f7328500bfda7ad7cd2b214bd5c47ef212c5af3243b0b4535c1 not found: ID does not exist"
Oct 30 22:08:28.735498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.735465    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6df5c77-jjjfr"]
Oct 30 22:08:28.739381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.739356    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c6df5c77-jjjfr"]
Oct 30 22:08:28.843355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.843271    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ghss5"]
Oct 30 22:08:28.847881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.847855    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ghss5"]
Oct 30 22:08:28.953397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.953363    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w857q"]
Oct 30 22:08:28.956888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:28.956862    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w857q"]
Oct 30 22:08:29.290241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:29.290209    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07787cb1-6a30-43a4-9a9f-270e03117bac" path="/var/lib/kubelet/pods/07787cb1-6a30-43a4-9a9f-270e03117bac/volumes"
Oct 30 22:08:29.290536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:29.290524    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3394b0f1-b32f-4548-8922-23292369e1cf" path="/var/lib/kubelet/pods/3394b0f1-b32f-4548-8922-23292369e1cf/volumes"
Oct 30 22:08:29.290832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:08:29.290820    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" path="/var/lib/kubelet/pods/7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d/volumes"
Oct 30 22:08:58.434213 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:08:58.434173    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:09:01.808352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:09:01.808263    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.468257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:09:16.468199    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:26.811911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:09:26.811871    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:28.435156 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:09:28.435120    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:09:58.436039 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:09:58.435997    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:10:00.342847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:00.342810    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:02.138812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:02.138778    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:03.989640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:03.989604    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:04.146984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:04.146952    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:22.754997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:22.754946    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:27.246081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:27.246036    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:28.436089 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:10:28.436039    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:10:31.190282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:31.190246    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:37.828086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828041    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"]
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828427    5028 cpu_manager.go:414] "containerMap: removing container" podUID="07787cb1-6a30-43a4-9a9f-270e03117bac" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828438    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07787cb1-6a30-43a4-9a9f-270e03117bac" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828457    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828462    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828487    5028 cpu_manager.go:414] "containerMap: removing container" podUID="3394b0f1-b32f-4548-8922-23292369e1cf" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828492    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3394b0f1-b32f-4548-8922-23292369e1cf" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828548    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="3394b0f1-b32f-4548-8922-23292369e1cf" containerName="authorino"
Oct 30 22:10:37.828553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828556    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="07787cb1-6a30-43a4-9a9f-270e03117bac" containerName="authorino"
Oct 30 22:10:37.828850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.828564    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae5fd05-1cb6-4b24-bc64-f2cbaf11918d" containerName="authorino"
Oct 30 22:10:37.831650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.831629    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:37.903493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.903449    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:37.931759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.931728    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dwlqg\""
Oct 30 22:10:37.931759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.931728    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 22:10:37.931759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.931753    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 22:10:37.932047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.931805    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 22:10:37.934059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.934034    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"]
Oct 30 22:10:37.991577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.991545    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:37.991577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.991578    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:37.991806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.991607    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:37.991806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.991747    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-kube-api-access-ndpmd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:37.991806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.991776    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:37.991902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:37.991865    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.092732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.092627    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.092883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.092665    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.092883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.092793    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.092883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.092845    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-kube-api-access-ndpmd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.092883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.092867    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.093075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.092930    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.093075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.093013    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.093200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.093069    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.093200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.093123    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.095186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.095160    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.095453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.095433    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.155356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.155321    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/9b57c28a-0d5d-4eac-bee0-7f18f78aac56-kube-api-access-ndpmd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t\" (UID: \"9b57c28a-0d5d-4eac-bee0-7f18f78aac56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.445251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.445162    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:38.817967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:38.817930    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"]
Oct 30 22:10:38.823533 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:10:38.823500    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b57c28a_0d5d_4eac_bee0_7f18f78aac56.slice/crio-165fd91bdbc9f4022baef3a0033b67467159e65952dce37c648339cb338cffbc WatchSource:0}: Error finding container 165fd91bdbc9f4022baef3a0033b67467159e65952dce37c648339cb338cffbc: Status 404 returned error can't find the container with id 165fd91bdbc9f4022baef3a0033b67467159e65952dce37c648339cb338cffbc
Oct 30 22:10:39.274015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:39.273976    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t" event={"ID":"9b57c28a-0d5d-4eac-bee0-7f18f78aac56","Type":"ContainerStarted","Data":"165fd91bdbc9f4022baef3a0033b67467159e65952dce37c648339cb338cffbc"}
Oct 30 22:10:41.290193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:41.290159    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t" event={"ID":"9b57c28a-0d5d-4eac-bee0-7f18f78aac56","Type":"ContainerStarted","Data":"dd9109d75c1cf0f67368a61f75a8907743cc9e8471edd4db16ae89787539f4bf"}
Oct 30 22:10:41.409602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:41.409543    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t" podStartSLOduration=2.871495667 podStartE2EDuration="4.409526236s" totalImagesPullingTime="1.538030569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:41.408537425 +0000 UTC m=+572.258050398" watchObservedRunningTime="2026-10-30 22:10:41.409526236 +0000 UTC m=+572.259039209"
Oct 30 22:10:41.445765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:41.445733    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:41.984165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:41.984125    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"]
Oct 30 22:10:41.987306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:41.987286    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.085064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.085035    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 22:10:42.087310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.087283    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"]
Oct 30 22:10:42.114435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.114403    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:42.129667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.129631    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.129837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.129701    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7eba18-77c0-4727-8a41-a77ee94779b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.129837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.129811    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.129923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.129861    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.129923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.129886    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.130055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.130035    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8hh\" (UniqueName: \"kubernetes.io/projected/5d7eba18-77c0-4727-8a41-a77ee94779b5-kube-api-access-vl8hh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.231590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.231545    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8hh\" (UniqueName: \"kubernetes.io/projected/5d7eba18-77c0-4727-8a41-a77ee94779b5-kube-api-access-vl8hh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.231842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.231618    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.231842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.231638    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7eba18-77c0-4727-8a41-a77ee94779b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.231842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.231752    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.231842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.231819    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.232034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.231847    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.232162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.232137    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.232227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.232210    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.232348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.232328    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.234140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.234112    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7eba18-77c0-4727-8a41-a77ee94779b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.234388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.234339    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7eba18-77c0-4727-8a41-a77ee94779b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.297363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.297325    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8hh\" (UniqueName: \"kubernetes.io/projected/5d7eba18-77c0-4727-8a41-a77ee94779b5-kube-api-access-vl8hh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk\" (UID: \"5d7eba18-77c0-4727-8a41-a77ee94779b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.299110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.299090    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:42.674758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:42.674664    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"]
Oct 30 22:10:42.679985 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:10:42.679951    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d7eba18_77c0_4727_8a41_a77ee94779b5.slice/crio-c172887c685f8916b17135d75002e645218d59e4203cbccd954f565c74aa82ec WatchSource:0}: Error finding container c172887c685f8916b17135d75002e645218d59e4203cbccd954f565c74aa82ec: Status 404 returned error can't find the container with id c172887c685f8916b17135d75002e645218d59e4203cbccd954f565c74aa82ec
Oct 30 22:10:43.305053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:43.305011    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" event={"ID":"5d7eba18-77c0-4727-8a41-a77ee94779b5","Type":"ContainerStarted","Data":"c172887c685f8916b17135d75002e645218d59e4203cbccd954f565c74aa82ec"}
Oct 30 22:10:44.312587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:44.312544    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" event={"ID":"5d7eba18-77c0-4727-8a41-a77ee94779b5","Type":"ContainerStarted","Data":"7449a6de63eb0e231f18cdf97d97f89b39eea5aff76cb73438bf2233818ce902"}
Oct 30 22:10:44.427681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:44.427624    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podStartSLOduration=2.792290268 podStartE2EDuration="3.427609496s" totalImagesPullingTime="635.319228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:44.425085975 +0000 UTC m=+575.274598949" watchObservedRunningTime="2026-10-30 22:10:44.427609496 +0000 UTC m=+575.277122469"
Oct 30 22:10:45.299436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:45.299402    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:45.854109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:45.854056    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:48.446065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:48.446024    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:48.460491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:48.460458    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:49.233303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:49.233224    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:49.350002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:49.349970    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t"
Oct 30 22:10:52.300099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:52.300055    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:52.312502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:52.312472    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:52.364498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:52.364470    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:10:53.587548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.587516    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"]
Oct 30 22:10:53.591133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.591112    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.641469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.641437    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ctw4\""
Oct 30 22:10:53.691416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.691383    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"]
Oct 30 22:10:53.739385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739347    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739383    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739410    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739520    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739561    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pln2f\" (UniqueName: \"kubernetes.io/projected/b3fe4656-786a-40d9-977c-83d2bbf357cd-kube-api-access-pln2f\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739580    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739632    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3fe4656-786a-40d9-977c-83d2bbf357cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739680    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.739818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.739730    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840489    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pln2f\" (UniqueName: \"kubernetes.io/projected/b3fe4656-786a-40d9-977c-83d2bbf357cd-kube-api-access-pln2f\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840526    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840547    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3fe4656-786a-40d9-977c-83d2bbf357cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840585    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840605    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840650    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840675    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.840910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840870    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.841183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.840932    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.841183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.841113    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.841291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.841271    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.841378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.841349    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.841506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.841413    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.841750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.841728    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3fe4656-786a-40d9-977c-83d2bbf357cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.843318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.843299    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.843445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.843428    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.902138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.902098    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3fe4656-786a-40d9-977c-83d2bbf357cd-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.902310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.902280    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pln2f\" (UniqueName: \"kubernetes.io/projected/b3fe4656-786a-40d9-977c-83d2bbf357cd-kube-api-access-pln2f\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wgcfm\" (UID: \"b3fe4656-786a-40d9-977c-83d2bbf357cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:53.904016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:53.904000    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:54.326788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:54.326750    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"]
Oct 30 22:10:54.332293 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:10:54.332264    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fe4656_786a_40d9_977c_83d2bbf357cd.slice/crio-3b9452a5af2fceb821eac2ad4a6c7ab2b4da25091584cc32b9d530ef23d62ebf WatchSource:0}: Error finding container 3b9452a5af2fceb821eac2ad4a6c7ab2b4da25091584cc32b9d530ef23d62ebf: Status 404 returned error can't find the container with id 3b9452a5af2fceb821eac2ad4a6c7ab2b4da25091584cc32b9d530ef23d62ebf
Oct 30 22:10:54.334801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:54.334768    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:54.334886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:54.334842    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:54.334886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:54.334871    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:54.364453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:54.364422    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" event={"ID":"b3fe4656-786a-40d9-977c-83d2bbf357cd","Type":"ContainerStarted","Data":"3b9452a5af2fceb821eac2ad4a6c7ab2b4da25091584cc32b9d530ef23d62ebf"}
Oct 30 22:10:54.728860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:54.728789    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:55.370656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:55.370613    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" event={"ID":"b3fe4656-786a-40d9-977c-83d2bbf357cd","Type":"ContainerStarted","Data":"7bca1d959c51ab1d48b8be8cd612897497d02260d757d8e9724e8fcef3250885"}
Oct 30 22:10:55.582802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:55.582746    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podStartSLOduration=2.582730939 podStartE2EDuration="2.582730939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:55.541414001 +0000 UTC m=+586.390926975" watchObservedRunningTime="2026-10-30 22:10:55.582730939 +0000 UTC m=+586.432243911"
Oct 30 22:10:55.904869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:55.904837    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:10:55.905890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:55.905869    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:10:55.905948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:55.905915    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 22:10:56.905348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:56.905306    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:10:56.905754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:56.905377    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 22:10:57.905138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:57.905098    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:10:57.905340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:57.905168    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 22:10:58.457371 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:10:58.457330    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:10:58.905368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:58.905326    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:10:58.905577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:58.905405    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 22:10:59.905241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:59.905198    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:10:59.905639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:10:59.905256    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 22:11:00.905384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:00.905338    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:11:00.905773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:00.905414    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 22:11:02.476656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:02.476615    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:02.905540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:02.905447    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:02.905540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:02.905519    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:03.946886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:03.946834    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:03.947383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:03.946894    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:04.075595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:04.074896    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:11:04.075595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:04.075327    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:11:04.076577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:04.076553    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm"
Oct 30 22:11:09.363622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:09.363589    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:11:09.364255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:09.364200    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:11:14.453307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.453265    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58989f478d-zkntj"]
Oct 30 22:11:14.458967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.458943    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.511418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.511383    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58989f478d-zkntj"]
Oct 30 22:11:14.628861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.628825    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"]
Oct 30 22:11:14.635978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.635949    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.643215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.643040    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/107baba0-bcf3-4809-9aab-fdbe79fdfa80-tls-cert\") pod \"authorino-58989f478d-zkntj\" (UID: \"107baba0-bcf3-4809-9aab-fdbe79fdfa80\") " pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.643215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.643172    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s878m\" (UniqueName: \"kubernetes.io/projected/107baba0-bcf3-4809-9aab-fdbe79fdfa80-kube-api-access-s878m\") pod \"authorino-58989f478d-zkntj\" (UID: \"107baba0-bcf3-4809-9aab-fdbe79fdfa80\") " pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.681730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.681662    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:14.686738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.686670    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 22:11:14.686904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.686709    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 22:11:14.686904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.686747    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 22:11:14.686904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.686789    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 22:11:14.686904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.686862    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-jzjjt\""
Oct 30 22:11:14.688037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.688013    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"]
Oct 30 22:11:14.743647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.743617    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/107baba0-bcf3-4809-9aab-fdbe79fdfa80-tls-cert\") pod \"authorino-58989f478d-zkntj\" (UID: \"107baba0-bcf3-4809-9aab-fdbe79fdfa80\") " pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.743830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.743668    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-hvpdx\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.743830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.743780    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fshb\" (UniqueName: \"kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb\") pod \"ai-gateway-controller-5d9cd99f45-hvpdx\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.743922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.743893    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s878m\" (UniqueName: \"kubernetes.io/projected/107baba0-bcf3-4809-9aab-fdbe79fdfa80-kube-api-access-s878m\") pod \"authorino-58989f478d-zkntj\" (UID: \"107baba0-bcf3-4809-9aab-fdbe79fdfa80\") " pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.746400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.746375    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/107baba0-bcf3-4809-9aab-fdbe79fdfa80-tls-cert\") pod \"authorino-58989f478d-zkntj\" (UID: \"107baba0-bcf3-4809-9aab-fdbe79fdfa80\") " pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.802659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.802633    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s878m\" (UniqueName: \"kubernetes.io/projected/107baba0-bcf3-4809-9aab-fdbe79fdfa80-kube-api-access-s878m\") pod \"authorino-58989f478d-zkntj\" (UID: \"107baba0-bcf3-4809-9aab-fdbe79fdfa80\") " pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:14.835409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.835376    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:14.844620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.844589    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fshb\" (UniqueName: \"kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb\") pod \"ai-gateway-controller-5d9cd99f45-hvpdx\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.844791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.844706    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-hvpdx\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.847149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.847126    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-hvpdx\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.903872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.903843    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fshb\" (UniqueName: \"kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb\") pod \"ai-gateway-controller-5d9cd99f45-hvpdx\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:14.950744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:14.950704    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:15.073791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:15.073758    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58989f478d-zkntj"
Oct 30 22:11:15.324043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:15.323961    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"]
Oct 30 22:11:15.329847 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:11:15.329819    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d02926c_51df_4614_a3ca_1212bdb7a5e3.slice/crio-862c60e96756d96f5084161e088eb0607c860463f6c409d740fa101c405ed357 WatchSource:0}: Error finding container 862c60e96756d96f5084161e088eb0607c860463f6c409d740fa101c405ed357: Status 404 returned error can't find the container with id 862c60e96756d96f5084161e088eb0607c860463f6c409d740fa101c405ed357
Oct 30 22:11:15.331313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:15.331287    5028 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:15.447679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:15.447637    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58989f478d-zkntj"]
Oct 30 22:11:15.454207 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:11:15.454169    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod107baba0_bcf3_4809_9aab_fdbe79fdfa80.slice/crio-3c5ac5dbdcb1ec680bca3ed018f2f9a3a50a140c2bddf63c34c1e49c9c6ef843 WatchSource:0}: Error finding container 3c5ac5dbdcb1ec680bca3ed018f2f9a3a50a140c2bddf63c34c1e49c9c6ef843: Status 404 returned error can't find the container with id 3c5ac5dbdcb1ec680bca3ed018f2f9a3a50a140c2bddf63c34c1e49c9c6ef843
Oct 30 22:11:15.495415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:15.495378    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58989f478d-zkntj" event={"ID":"107baba0-bcf3-4809-9aab-fdbe79fdfa80","Type":"ContainerStarted","Data":"3c5ac5dbdcb1ec680bca3ed018f2f9a3a50a140c2bddf63c34c1e49c9c6ef843"}
Oct 30 22:11:15.496536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:15.496511    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx" event={"ID":"5d02926c-51df-4614-a3ca-1212bdb7a5e3","Type":"ContainerStarted","Data":"862c60e96756d96f5084161e088eb0607c860463f6c409d740fa101c405ed357"}
Oct 30 22:11:17.509439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:17.509396    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58989f478d-zkntj" event={"ID":"107baba0-bcf3-4809-9aab-fdbe79fdfa80","Type":"ContainerStarted","Data":"477110abdc905d594d07f190bd17607dc3e4be9535e8eaa672264917b8e7bb91"}
Oct 30 22:11:17.620663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:17.620604    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58989f478d-zkntj" podStartSLOduration=1.893454131 podStartE2EDuration="3.620589156s" totalImagesPullingTime="1.727135025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:17.619394338 +0000 UTC m=+608.468907313" watchObservedRunningTime="2026-10-30 22:11:17.620589156 +0000 UTC m=+608.470102129"
Oct 30 22:11:17.650863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:17.650827    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ff796694c-s4bfp"]
Oct 30 22:11:17.651114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:17.651086    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6ff796694c-s4bfp" podUID="6be15d82-f54d-4237-8508-2b09ca5c4d8a" containerName="authorino" containerID="cri-o://de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379" gracePeriod=30
Oct 30 22:11:17.944747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:17.944714    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:11:18.077473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.077385    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert\" (UniqueName: \"kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert\") pod \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") "
Oct 30 22:11:18.077473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.077462    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr\" (UniqueName: \"kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr\") pod \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\" (UID: \"6be15d82-f54d-4237-8508-2b09ca5c4d8a\") "
Oct 30 22:11:18.079575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.079543    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr" pod "6be15d82-f54d-4237-8508-2b09ca5c4d8a" (UID: "6be15d82-f54d-4237-8508-2b09ca5c4d8a"). InnerVolumeSpecName "kube-api-access-z87gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:18.090444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.090409    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert" pod "6be15d82-f54d-4237-8508-2b09ca5c4d8a" (UID: "6be15d82-f54d-4237-8508-2b09ca5c4d8a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:18.179159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.179121    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87gr\" (UniqueName: \"kubernetes.io/projected/6be15d82-f54d-4237-8508-2b09ca5c4d8a-kube-api-access-z87gr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:18.179159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.179159    5028 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6be15d82-f54d-4237-8508-2b09ca5c4d8a-tls-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:18.518443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.518406    5028 generic.go:356] "Generic (PLEG): container finished" podID="6be15d82-f54d-4237-8508-2b09ca5c4d8a" containerID="de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379" exitCode=0
Oct 30 22:11:18.518961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.518480    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ff796694c-s4bfp"
Oct 30 22:11:18.518961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.518490    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ff796694c-s4bfp" event={"ID":"6be15d82-f54d-4237-8508-2b09ca5c4d8a","Type":"ContainerDied","Data":"de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379"}
Oct 30 22:11:18.518961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.518552    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ff796694c-s4bfp" event={"ID":"6be15d82-f54d-4237-8508-2b09ca5c4d8a","Type":"ContainerDied","Data":"20c43bddd0374ca62c5c3c5d6340c43ac42f22712c486c5d966d109a0f77cfd0"}
Oct 30 22:11:18.518961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.518574    5028 scope.go:122] "RemoveContainer" containerID="de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379"
Oct 30 22:11:18.529364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.529347    5028 scope.go:122] "RemoveContainer" containerID="de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379"
Oct 30 22:11:18.529666 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:11:18.529650    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379\": container with ID starting with de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379 not found: ID does not exist" containerID="de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379"
Oct 30 22:11:18.529831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.529676    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379"} err="failed to get container status \"de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379\": rpc error: code = NotFound desc = could not find container \"de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379\": container with ID starting with de970646b760cbf9cd328a92c24562a97c5fab17fedcbbb074b9c2cbee03c379 not found: ID does not exist"
Oct 30 22:11:18.677261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.677233    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6ff796694c-s4bfp"]
Oct 30 22:11:18.681347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:18.681325    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6ff796694c-s4bfp"]
Oct 30 22:11:19.290022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:19.289988    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be15d82-f54d-4237-8508-2b09ca5c4d8a" path="/var/lib/kubelet/pods/6be15d82-f54d-4237-8508-2b09ca5c4d8a/volumes"
Oct 30 22:11:19.527529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:19.527491    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx" event={"ID":"5d02926c-51df-4614-a3ca-1212bdb7a5e3","Type":"ContainerStarted","Data":"45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df"}
Oct 30 22:11:19.528044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:19.527578    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:19.642926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:19.642816    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx" podStartSLOduration=2.3651719 podStartE2EDuration="5.642795158s" totalImagesPullingTime="3.277623258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:19.642666675 +0000 UTC m=+610.492179650" watchObservedRunningTime="2026-10-30 22:11:19.642795158 +0000 UTC m=+610.492308135"
Oct 30 22:11:22.476970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.476921    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:22.651020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.650983    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:22.881679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.881597    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"]
Oct 30 22:11:22.882021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.882007    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6be15d82-f54d-4237-8508-2b09ca5c4d8a" containerName="authorino"
Oct 30 22:11:22.882071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.882023    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be15d82-f54d-4237-8508-2b09ca5c4d8a" containerName="authorino"
Oct 30 22:11:22.882115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.882106    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be15d82-f54d-4237-8508-2b09ca5c4d8a" containerName="authorino"
Oct 30 22:11:22.884895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.884876    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:22.927490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.927454    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739abe96-0923-4b32-8e9b-c4e66f7237da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t9lxh\" (UID: \"739abe96-0923-4b32-8e9b-c4e66f7237da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:22.927697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.927497    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfq6\" (UniqueName: \"kubernetes.io/projected/739abe96-0923-4b32-8e9b-c4e66f7237da-kube-api-access-lmfq6\") pod \"payload-pre-processing-7b98bb678c-t9lxh\" (UID: \"739abe96-0923-4b32-8e9b-c4e66f7237da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:22.940764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.940729    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"]
Oct 30 22:11:22.984430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.984395    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 22:11:22.984635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:22.984581    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2fsrc\""
Oct 30 22:11:23.028230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.028191    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739abe96-0923-4b32-8e9b-c4e66f7237da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t9lxh\" (UID: \"739abe96-0923-4b32-8e9b-c4e66f7237da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:23.028485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.028452    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfq6\" (UniqueName: \"kubernetes.io/projected/739abe96-0923-4b32-8e9b-c4e66f7237da-kube-api-access-lmfq6\") pod \"payload-pre-processing-7b98bb678c-t9lxh\" (UID: \"739abe96-0923-4b32-8e9b-c4e66f7237da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:23.029138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.029114    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739abe96-0923-4b32-8e9b-c4e66f7237da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-t9lxh\" (UID: \"739abe96-0923-4b32-8e9b-c4e66f7237da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:23.033912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.033891    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fdzns"]
Oct 30 22:11:23.037611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.037597    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.083454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.083422    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fdzns"]
Oct 30 22:11:23.089137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.089107    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfq6\" (UniqueName: \"kubernetes.io/projected/739abe96-0923-4b32-8e9b-c4e66f7237da-kube-api-access-lmfq6\") pod \"payload-pre-processing-7b98bb678c-t9lxh\" (UID: \"739abe96-0923-4b32-8e9b-c4e66f7237da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:23.129248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.129212    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpgl\" (UniqueName: \"kubernetes.io/projected/7c08f8f7-8f06-4532-8c79-43d5d77ce9dc-kube-api-access-dtpgl\") pod \"payload-processing-55fbbbf9dc-fdzns\" (UID: \"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.129408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.129264    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c08f8f7-8f06-4532-8c79-43d5d77ce9dc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fdzns\" (UID: \"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.198466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.198372    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:23.230082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.230048    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c08f8f7-8f06-4532-8c79-43d5d77ce9dc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fdzns\" (UID: \"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.230250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.230160    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpgl\" (UniqueName: \"kubernetes.io/projected/7c08f8f7-8f06-4532-8c79-43d5d77ce9dc-kube-api-access-dtpgl\") pod \"payload-processing-55fbbbf9dc-fdzns\" (UID: \"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.230799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.230778    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c08f8f7-8f06-4532-8c79-43d5d77ce9dc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fdzns\" (UID: \"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.289416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.289381    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpgl\" (UniqueName: \"kubernetes.io/projected/7c08f8f7-8f06-4532-8c79-43d5d77ce9dc-kube-api-access-dtpgl\") pod \"payload-processing-55fbbbf9dc-fdzns\" (UID: \"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.351469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.351419    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:23.570944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.570909    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"]
Oct 30 22:11:23.576111 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:11:23.576070    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739abe96_0923_4b32_8e9b_c4e66f7237da.slice/crio-fea5229b5d9f3fcd9cb682f81e1c18858601b30410cc040916916731ca2d7e50 WatchSource:0}: Error finding container fea5229b5d9f3fcd9cb682f81e1c18858601b30410cc040916916731ca2d7e50: Status 404 returned error can't find the container with id fea5229b5d9f3fcd9cb682f81e1c18858601b30410cc040916916731ca2d7e50
Oct 30 22:11:23.727735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.727702    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fdzns"]
Oct 30 22:11:23.733843 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:11:23.733810    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c08f8f7_8f06_4532_8c79_43d5d77ce9dc.slice/crio-3b878db4eab3367930da07c152cb5c4997e99871f5763d6d0ba9ffd6f3c9508c WatchSource:0}: Error finding container 3b878db4eab3367930da07c152cb5c4997e99871f5763d6d0ba9ffd6f3c9508c: Status 404 returned error can't find the container with id 3b878db4eab3367930da07c152cb5c4997e99871f5763d6d0ba9ffd6f3c9508c
Oct 30 22:11:23.881882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:23.881800    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:24.560915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:24.560878    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh" event={"ID":"739abe96-0923-4b32-8e9b-c4e66f7237da","Type":"ContainerStarted","Data":"fea5229b5d9f3fcd9cb682f81e1c18858601b30410cc040916916731ca2d7e50"}
Oct 30 22:11:24.562204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:24.562169    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns" event={"ID":"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc","Type":"ContainerStarted","Data":"3b878db4eab3367930da07c152cb5c4997e99871f5763d6d0ba9ffd6f3c9508c"}
Oct 30 22:11:27.582426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:27.582370    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh" event={"ID":"739abe96-0923-4b32-8e9b-c4e66f7237da","Type":"ContainerStarted","Data":"d39d2d6aa9af3ea023ab787408799842ae8e25fb54a6edffa4fd25132911e439"}
Oct 30 22:11:27.583084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:27.582481    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:27.584340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:27.584314    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns" event={"ID":"7c08f8f7-8f06-4532-8c79-43d5d77ce9dc","Type":"ContainerStarted","Data":"9af26af3e2347225899fa7d834a26a92594dce2bc2c15d1d28923cb61487cdff"}
Oct 30 22:11:27.584481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:27.584408    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:27.704440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:27.704383    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh" podStartSLOduration=2.512315536 podStartE2EDuration="5.704369065s" totalImagesPullingTime="3.192053529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:27.702618404 +0000 UTC m=+618.552131376" watchObservedRunningTime="2026-10-30 22:11:27.704369065 +0000 UTC m=+618.553882038"
Oct 30 22:11:27.818011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:27.817958    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns" podStartSLOduration=2.731462479 podStartE2EDuration="5.817943445s" totalImagesPullingTime="3.086480966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:27.814923648 +0000 UTC m=+618.664436636" watchObservedRunningTime="2026-10-30 22:11:27.817943445 +0000 UTC m=+618.667456434"
Oct 30 22:11:28.435319 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:11:28.435276    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:11:30.538620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:30.538592    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:30.901661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:30.901581    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"]
Oct 30 22:11:30.905884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:30.905867    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:30.964024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:30.963994    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"]
Oct 30 22:11:31.002268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.002237    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwd2\" (UniqueName: \"kubernetes.io/projected/8eed38c4-f9c7-4635-925c-0b5cc8e51ebd-kube-api-access-bkwd2\") pod \"ai-gateway-controller-7cd4b5bfc5-nqqbs\" (UID: \"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd\") " pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.002411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.002369    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eed38c4-f9c7-4635-925c-0b5cc8e51ebd-metrics-certs\") pod \"ai-gateway-controller-7cd4b5bfc5-nqqbs\" (UID: \"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd\") " pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.103185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.103150    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eed38c4-f9c7-4635-925c-0b5cc8e51ebd-metrics-certs\") pod \"ai-gateway-controller-7cd4b5bfc5-nqqbs\" (UID: \"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd\") " pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.103376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.103249    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwd2\" (UniqueName: \"kubernetes.io/projected/8eed38c4-f9c7-4635-925c-0b5cc8e51ebd-kube-api-access-bkwd2\") pod \"ai-gateway-controller-7cd4b5bfc5-nqqbs\" (UID: \"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd\") " pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.105564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.105542    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eed38c4-f9c7-4635-925c-0b5cc8e51ebd-metrics-certs\") pod \"ai-gateway-controller-7cd4b5bfc5-nqqbs\" (UID: \"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd\") " pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.162862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.162790    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwd2\" (UniqueName: \"kubernetes.io/projected/8eed38c4-f9c7-4635-925c-0b5cc8e51ebd-kube-api-access-bkwd2\") pod \"ai-gateway-controller-7cd4b5bfc5-nqqbs\" (UID: \"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd\") " pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.218798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.218767    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:31.591898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.591860    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"]
Oct 30 22:11:31.597323 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:11:31.597289    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eed38c4_f9c7_4635_925c_0b5cc8e51ebd.slice/crio-b7fc0846e6af25fc672271b79fdfcf7ea695c608c87d210b64c92939c671dea3 WatchSource:0}: Error finding container b7fc0846e6af25fc672271b79fdfcf7ea695c608c87d210b64c92939c671dea3: Status 404 returned error can't find the container with id b7fc0846e6af25fc672271b79fdfcf7ea695c608c87d210b64c92939c671dea3
Oct 30 22:11:31.607083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:31.607046    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs" event={"ID":"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd","Type":"ContainerStarted","Data":"b7fc0846e6af25fc672271b79fdfcf7ea695c608c87d210b64c92939c671dea3"}
Oct 30 22:11:32.614059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:32.614022    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs" event={"ID":"8eed38c4-f9c7-4635-925c-0b5cc8e51ebd","Type":"ContainerStarted","Data":"54684a0342d63f43b88c25dd10f589cd90abd180d0f04b4935083239075b5c0b"}
Oct 30 22:11:32.614455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:32.614093    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:32.731355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:32.731255    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs" podStartSLOduration=1.850628806 podStartE2EDuration="2.731240884s" totalImagesPullingTime="880.612078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:32.730379387 +0000 UTC m=+623.579892361" watchObservedRunningTime="2026-10-30 22:11:32.731240884 +0000 UTC m=+623.580753857"
Oct 30 22:11:38.593155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:38.593126    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fdzns"
Oct 30 22:11:38.593557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:38.593194    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-t9lxh"
Oct 30 22:11:43.620953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:43.620918    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cd4b5bfc5-nqqbs"
Oct 30 22:11:43.766205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:43.766171    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"]
Oct 30 22:11:43.766453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:43.766422    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx" podUID="5d02926c-51df-4614-a3ca-1212bdb7a5e3" containerName="manager" containerID="cri-o://45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df" gracePeriod=30
Oct 30 22:11:44.013247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.013224    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:44.117076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.117040    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb\" (UniqueName: \"kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb\") pod \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") "
Oct 30 22:11:44.117275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.117130    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs\") pod \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\" (UID: \"5d02926c-51df-4614-a3ca-1212bdb7a5e3\") "
Oct 30 22:11:44.119271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.119239    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs" pod "5d02926c-51df-4614-a3ca-1212bdb7a5e3" (UID: "5d02926c-51df-4614-a3ca-1212bdb7a5e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:44.119387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.119321    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb" pod "5d02926c-51df-4614-a3ca-1212bdb7a5e3" (UID: "5d02926c-51df-4614-a3ca-1212bdb7a5e3"). InnerVolumeSpecName "kube-api-access-7fshb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:44.218497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.218469    5028 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d02926c-51df-4614-a3ca-1212bdb7a5e3-metrics-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:44.218497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.218497    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fshb\" (UniqueName: \"kubernetes.io/projected/5d02926c-51df-4614-a3ca-1212bdb7a5e3-kube-api-access-7fshb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:44.677077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.676992    5028 generic.go:356] "Generic (PLEG): container finished" podID="5d02926c-51df-4614-a3ca-1212bdb7a5e3" containerID="45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df" exitCode=0
Oct 30 22:11:44.677077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.677038    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"
Oct 30 22:11:44.677557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.677072    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx" event={"ID":"5d02926c-51df-4614-a3ca-1212bdb7a5e3","Type":"ContainerDied","Data":"45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df"}
Oct 30 22:11:44.677557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.677123    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx" event={"ID":"5d02926c-51df-4614-a3ca-1212bdb7a5e3","Type":"ContainerDied","Data":"862c60e96756d96f5084161e088eb0607c860463f6c409d740fa101c405ed357"}
Oct 30 22:11:44.677557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.677145    5028 scope.go:122] "RemoveContainer" containerID="45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df"
Oct 30 22:11:44.687123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.687106    5028 scope.go:122] "RemoveContainer" containerID="45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df"
Oct 30 22:11:44.687377 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:11:44.687358    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df\": container with ID starting with 45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df not found: ID does not exist" containerID="45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df"
Oct 30 22:11:44.687427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.687387    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df"} err="failed to get container status \"45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df\": rpc error: code = NotFound desc = could not find container \"45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df\": container with ID starting with 45a7ac44e846b86e95bb9f3850188960d7e9bb4c23c5548d3a8bc48ca5ef03df not found: ID does not exist"
Oct 30 22:11:44.835087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.835052    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"]
Oct 30 22:11:44.882412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:44.882370    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-hvpdx"]
Oct 30 22:11:45.290047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:45.290015    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d02926c-51df-4614-a3ca-1212bdb7a5e3" path="/var/lib/kubelet/pods/5d02926c-51df-4614-a3ca-1212bdb7a5e3/volumes"
Oct 30 22:11:45.457096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:45.457061    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:45.976735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:45.976696    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:47.608373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:47.608335    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:47.763436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:47.763404    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:49.238873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:49.238840    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:49.400452 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:49.400418    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:54.143362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:54.143317    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:54.295758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:11:54.295730    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.434607 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:11:58.434568    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:12:02.801954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:02.801882    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:13.834792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:13.834719    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.008680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.008640    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.384487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.384449    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"]
Oct 30 22:12:14.384916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.384900    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5d02926c-51df-4614-a3ca-1212bdb7a5e3" containerName="manager"
Oct 30 22:12:14.384916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.384914    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d02926c-51df-4614-a3ca-1212bdb7a5e3" containerName="manager"
Oct 30 22:12:14.385013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.384989    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d02926c-51df-4614-a3ca-1212bdb7a5e3" containerName="manager"
Oct 30 22:12:14.391669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.391648    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393283    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393340    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393386    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393514    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393570    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393622    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393674    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393730    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhnj\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.393803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.393785    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.446381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.446344    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-af30a9-openshift-default-dockercfg-fzzwk\""
Oct 30 22:12:14.494566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494533    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.494768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494575    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.494768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494617    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.494768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494644    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.494768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494668    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhnj\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.494768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494729    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494812    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494840    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.494863    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.495218    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.495417    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.495477    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.495678    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.495815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.495761    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.496024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.496003    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"]
Oct 30 22:12:14.497230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.497205    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.497388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.497323    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.497658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.497634    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.557542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.557510    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.557715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.557548    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhnj\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.679947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.679874    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:14.705852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.705818    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:14.827715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:14.827656    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.003118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.003080    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"]
Oct 30 22:12:15.008056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.008031    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.058959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.058931    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eb4aa1-openshift-default-dockercfg-l8phj\""
Oct 30 22:12:15.064794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.064759    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.073551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.073530    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"]
Oct 30 22:12:15.100857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.100821    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.100988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.100861    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.100988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.100887    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkw8g\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.101061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.101035    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.101100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.101073    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.101146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.101096    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.101194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.101178    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.101239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.101209    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.101297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.101275    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.170654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.170621    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"]
Oct 30 22:12:15.176541 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:15.176512    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode660d4aa_1866_4615_a66b_b5d3d499d00b.slice/crio-dad44ae6d9d8a23116c7bdb5351179bf4176440ba9d07da43ce0fc9b7f2d2488 WatchSource:0}: Error finding container dad44ae6d9d8a23116c7bdb5351179bf4176440ba9d07da43ce0fc9b7f2d2488: Status 404 returned error can't find the container with id dad44ae6d9d8a23116c7bdb5351179bf4176440ba9d07da43ce0fc9b7f2d2488
Oct 30 22:12:15.179079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.179047    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:15.179178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.179123    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:15.179178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.179167    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:15.202512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202490    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202524    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202544    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202566    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202592    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202617    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202633    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkw8g\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202716    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202757    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.202947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.202925    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.203023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.203001    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.203089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.203061    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.203145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.203126    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.203359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.203341    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.204778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.204758    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.205042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.205026    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.235495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.235473    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.261553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.261490    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.261840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.261818    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkw8g\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.297740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.297707    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"]
Oct 30 22:12:15.301522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.301499    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.322379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.322346    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:15.352790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.352346    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-23f3d4-openshift-default-dockercfg-c5fxn\""
Oct 30 22:12:15.361634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.361608    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"]
Oct 30 22:12:15.392448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.392327    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.408434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.408399    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409460    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcvk\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409552    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409581    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409617    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409646    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409708    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409792    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.409921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.409817    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.511311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.511247    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.511938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.511481    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.511938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.511516    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.511938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.511595    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.512475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.512029    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.512475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.512178    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcvk\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.512475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.512252    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.512475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.512300    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.512475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.512328    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.512475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.512361    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.513280 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.513243    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.514174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.513947    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.514287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.514267    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.514507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.514489    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.515248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.515229    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.518394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.518364    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.551808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.551775    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.571261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.571231    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.574667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.574647    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcvk\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.619225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.618842    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:15.704025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.703994    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.731107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.731078    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"]
Oct 30 22:12:15.736540 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:15.736514    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9a5089_8749_4d9b_b53d_96eb145c757a.slice/crio-a69322ce1f4c89153c7959ce2201c1cffa2e1cf2edf557262c62057cf6d6ebea WatchSource:0}: Error finding container a69322ce1f4c89153c7959ce2201c1cffa2e1cf2edf557262c62057cf6d6ebea: Status 404 returned error can't find the container with id a69322ce1f4c89153c7959ce2201c1cffa2e1cf2edf557262c62057cf6d6ebea
Oct 30 22:12:15.738907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.738874    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:15.739004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.738946    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:15.739004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.738975    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:15.840597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.840557    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" event={"ID":"e660d4aa-1866-4615-a66b-b5d3d499d00b","Type":"ContainerStarted","Data":"32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"}
Oct 30 22:12:15.840805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.840620    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" event={"ID":"e660d4aa-1866-4615-a66b-b5d3d499d00b","Type":"ContainerStarted","Data":"dad44ae6d9d8a23116c7bdb5351179bf4176440ba9d07da43ce0fc9b7f2d2488"}
Oct 30 22:12:15.842411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.842382    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" event={"ID":"cd9a5089-8749-4d9b-b53d-96eb145c757a","Type":"ContainerStarted","Data":"9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"}
Oct 30 22:12:15.842563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.842424    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" event={"ID":"cd9a5089-8749-4d9b-b53d-96eb145c757a","Type":"ContainerStarted","Data":"a69322ce1f4c89153c7959ce2201c1cffa2e1cf2edf557262c62057cf6d6ebea"}
Oct 30 22:12:15.852802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.852768    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.956993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:15.956943    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podStartSLOduration=1.9569278300000001 podStartE2EDuration="1.95692783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:15.955710647 +0000 UTC m=+666.805223610" watchObservedRunningTime="2026-10-30 22:12:15.95692783 +0000 UTC m=+666.806440803"
Oct 30 22:12:16.058935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.058848    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:16.067406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.067372    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"]
Oct 30 22:12:16.073444 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:16.073420    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd86ee34_1f3f_4bf6_b93c_ccb204fdf1a9.slice/crio-3b54ae801323717b953164a774d9112f7dd1cddf1a27c49ce8a0a363c9dbdb52 WatchSource:0}: Error finding container 3b54ae801323717b953164a774d9112f7dd1cddf1a27c49ce8a0a363c9dbdb52: Status 404 returned error can't find the container with id 3b54ae801323717b953164a774d9112f7dd1cddf1a27c49ce8a0a363c9dbdb52
Oct 30 22:12:16.076386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.076343    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:16.076493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.076445    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:16.076556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.076503    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:16.089186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.089136    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podStartSLOduration=2.089121757 podStartE2EDuration="2.089121757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:16.085748023 +0000 UTC m=+666.935260987" watchObservedRunningTime="2026-10-30 22:12:16.089121757 +0000 UTC m=+666.938634730"
Oct 30 22:12:16.219335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.219298    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"]
Oct 30 22:12:16.223907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.223886    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.236436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.236395    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:16.274377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.274345    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d2d4c3-openshift-default-dockercfg-kzbw6\""
Oct 30 22:12:16.283224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.283195    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"]
Oct 30 22:12:16.320981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.320900    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.320982    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321061    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321102    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321271    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321301    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321331    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321434    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fgx\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.321525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.321487    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.323681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.323666    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:16.328624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.328602    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:16.398546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.398496    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:16.422532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422498    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422537    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422554    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422575    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422605    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422652    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422679    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fgx\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422724    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.422978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.422806    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.423053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.423029    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.423112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.423048    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.423163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.423141    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.423473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.423444    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.423593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.423474    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.425151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.425127    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.425293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.425275    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.481905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.481873    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.482044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.481943    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fgx\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.539350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.539314    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:16.559528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.559472    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:16.706222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.706176    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:16.711774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.711751    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:16.849186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.849094    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" event={"ID":"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9","Type":"ContainerStarted","Data":"a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"}
Oct 30 22:12:16.849186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.849155    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" event={"ID":"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9","Type":"ContainerStarted","Data":"3b54ae801323717b953164a774d9112f7dd1cddf1a27c49ce8a0a363c9dbdb52"}
Oct 30 22:12:16.849829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.849500    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:16.849829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.849547    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:16.850277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.850253    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:16.850381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.850310    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:16.954613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.954577    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"]
Oct 30 22:12:16.959868 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:16.959842    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb242cced_73d5_48cf_9819_84803277f9da.slice/crio-3a11c5df6190935d3ae73ce747422ba9041e138754db56a0c0077aefba7eaf8a WatchSource:0}: Error finding container 3a11c5df6190935d3ae73ce747422ba9041e138754db56a0c0077aefba7eaf8a: Status 404 returned error can't find the container with id 3a11c5df6190935d3ae73ce747422ba9041e138754db56a0c0077aefba7eaf8a
Oct 30 22:12:16.962548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.962518    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:16.962622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.962588    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:16.962622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.962614    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:16.968095 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:16.968032    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podStartSLOduration=1.968020687 podStartE2EDuration="1.968020687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:16.966246142 +0000 UTC m=+667.815759113" watchObservedRunningTime="2026-10-30 22:12:16.968020687 +0000 UTC m=+667.817533669"
Oct 30 22:12:17.620409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.620374    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:17.628357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.628329    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:17.855949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.855909    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" event={"ID":"b242cced-73d5-48cf-9819-84803277f9da","Type":"ContainerStarted","Data":"a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"}
Oct 30 22:12:17.856165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.855969    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" event={"ID":"b242cced-73d5-48cf-9819-84803277f9da","Type":"ContainerStarted","Data":"3a11c5df6190935d3ae73ce747422ba9041e138754db56a0c0077aefba7eaf8a"}
Oct 30 22:12:17.856479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.856461    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:17.857380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.857352    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:17.973544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:17.973490    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podStartSLOduration=1.973475629 podStartE2EDuration="1.973475629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:17.971758883 +0000 UTC m=+668.821271855" watchObservedRunningTime="2026-10-30 22:12:17.973475629 +0000 UTC m=+668.822988600"
Oct 30 22:12:18.540135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:18.540100    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:18.546099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:18.546068    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:18.862042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:18.861959    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:18.863092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:18.863075    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:12:19.324919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:19.324882    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:19.505517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:19.505463    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:19.666250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:19.666149    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:19.995589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:19.995554    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:20.160468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:20.160418    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:20.315300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:20.315215    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:20.803951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:20.803917    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:20.998542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:20.998510    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:21.153870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:21.153786    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:21.573577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:21.573543    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:21.776413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:21.776379    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:21.926794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:21.926705    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:22.245794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:22.245757    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:22.419171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:22.419137    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:22.581889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:22.581800    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:24.512292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:24.512258    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:24.672412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:24.672376    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:24.924265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:24.924193    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"]
Oct 30 22:12:24.928063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:24.928044    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:24.987149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:24.987110    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"]
Oct 30 22:12:25.017670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.017633    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.017860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.017708    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrhk\" (UniqueName: \"kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk\") pod \"payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.027466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.027438    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b13755\""
Oct 30 22:12:25.045050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.045010    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b13755-dockercfg-s9sx2\""
Oct 30 22:12:25.077464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.077431    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"]
Oct 30 22:12:25.081175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.081155    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.119253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.119223    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.119382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.119279    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrhk\" (UniqueName: \"kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk\") pod \"payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.120048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.120026    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.124338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.124313    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"]
Oct 30 22:12:25.178206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.178135    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrhk\" (UniqueName: \"kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk\") pod \"payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.220176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.220145    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxf5x\" (UniqueName: \"kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x\") pod \"payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.220351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.220192    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.241275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.241247    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.321315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.321284    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxf5x\" (UniqueName: \"kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x\") pod \"payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.321482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.321324    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.322060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.322041    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.325422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.325395    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:25.380137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.380104    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxf5x\" (UniqueName: \"kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x\") pod \"payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.395056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.395026    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.544394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.544340    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:25.619727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.619680    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"]
Oct 30 22:12:25.625586 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:25.625563    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08508b9b_d3ed_42c8_aedd_13aaad6cb943.slice/crio-9e8b7d6387a350694daf04eb0e2f24f68407b464cdff31c61ad29e2c1f203557 WatchSource:0}: Error finding container 9e8b7d6387a350694daf04eb0e2f24f68407b464cdff31c61ad29e2c1f203557: Status 404 returned error can't find the container with id 9e8b7d6387a350694daf04eb0e2f24f68407b464cdff31c61ad29e2c1f203557
Oct 30 22:12:25.722786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.722755    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:25.774902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.774871    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"]
Oct 30 22:12:25.780945 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:25.780909    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a00ab_df8f_4571_b99d_1b879a9c323f.slice/crio-d904ce228232eeb85a8adff90018722dca90665b61585d41e697f3fc9f808cfa WatchSource:0}: Error finding container d904ce228232eeb85a8adff90018722dca90665b61585d41e697f3fc9f808cfa: Status 404 returned error can't find the container with id d904ce228232eeb85a8adff90018722dca90665b61585d41e697f3fc9f808cfa
Oct 30 22:12:25.908262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.908229    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" event={"ID":"08508b9b-d3ed-42c8-aedd-13aaad6cb943","Type":"ContainerStarted","Data":"4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800"}
Oct 30 22:12:25.908421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.908296    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:25.908421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.908319    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" event={"ID":"08508b9b-d3ed-42c8-aedd-13aaad6cb943","Type":"ContainerStarted","Data":"9e8b7d6387a350694daf04eb0e2f24f68407b464cdff31c61ad29e2c1f203557"}
Oct 30 22:12:25.910089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.910059    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg" event={"ID":"ce9a00ab-df8f-4571-b99d-1b879a9c323f","Type":"ContainerStarted","Data":"d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698"}
Oct 30 22:12:25.910194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.910119    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg" event={"ID":"ce9a00ab-df8f-4571-b99d-1b879a9c323f","Type":"ContainerStarted","Data":"d904ce228232eeb85a8adff90018722dca90665b61585d41e697f3fc9f808cfa"}
Oct 30 22:12:25.910269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.910221    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:25.994944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:25.994909    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.020733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:26.020646    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" podStartSLOduration=2.02062856 podStartE2EDuration="2.02062856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:26.020166889 +0000 UTC m=+676.869679863" watchObservedRunningTime="2026-10-30 22:12:26.02062856 +0000 UTC m=+676.870141537"
Oct 30 22:12:26.133702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:26.133635    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg" podStartSLOduration=1.133622645 podStartE2EDuration="1.133622645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:26.13205357 +0000 UTC m=+676.981566542" watchObservedRunningTime="2026-10-30 22:12:26.133622645 +0000 UTC m=+676.983135618"
Oct 30 22:12:26.148978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:26.148947    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:26.311950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:26.311856    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.444254 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:12:28.444217    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:12:29.165625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.165586    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:29.329499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.329462    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:29.562964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.562926    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"]
Oct 30 22:12:29.567057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.567034    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.665521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.665482    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.665716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.665545    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s82t\" (UniqueName: \"kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t\") pod \"payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.682381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.682350    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-af30a9\""
Oct 30 22:12:29.682381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.682358    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-af30a9-dockercfg-5lr8k\""
Oct 30 22:12:29.685031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.685000    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"]
Oct 30 22:12:29.732343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.732311    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"]
Oct 30 22:12:29.736203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.736186    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:29.766937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.766904    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.767084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.766959    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s82t\" (UniqueName: \"kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t\") pod \"payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.767614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.767592    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.788980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.788950    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"]
Oct 30 22:12:29.828558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.828487    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s82t\" (UniqueName: \"kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t\") pod \"payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.868031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.867995    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflvc\" (UniqueName: \"kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc\") pod \"payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:29.868212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.868149    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:29.880814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.880782    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:29.969346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.969308    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:29.969503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.969409    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflvc\" (UniqueName: \"kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc\") pod \"payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:29.970348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:29.970296    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:30.028601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.028567    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflvc\" (UniqueName: \"kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc\") pod \"payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:30.049484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.049444    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:30.258501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.258466    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"]
Oct 30 22:12:30.263970 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:30.263942    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2368593a_b34f_4133_b1cd_1e09488a8ca0.slice/crio-40833ef72bf7eb9ff28af42fc38409f0734d06e0bf117a83fe0d44013a491d2d WatchSource:0}: Error finding container 40833ef72bf7eb9ff28af42fc38409f0734d06e0bf117a83fe0d44013a491d2d: Status 404 returned error can't find the container with id 40833ef72bf7eb9ff28af42fc38409f0734d06e0bf117a83fe0d44013a491d2d
Oct 30 22:12:30.423636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.423562    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"]
Oct 30 22:12:30.428927 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:30.428886    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8a636e_75a3_4da7_8ccf_058f43de40aa.slice/crio-49047c76cd06ec5fc4280ba58fd982cdc5c324fed798540d7f178e4ae4ddc07e WatchSource:0}: Error finding container 49047c76cd06ec5fc4280ba58fd982cdc5c324fed798540d7f178e4ae4ddc07e: Status 404 returned error can't find the container with id 49047c76cd06ec5fc4280ba58fd982cdc5c324fed798540d7f178e4ae4ddc07e
Oct 30 22:12:30.717959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.717926    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:30.913374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.913335    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:30.940930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.940895    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" event={"ID":"3d8a636e-75a3-4da7-8ccf-058f43de40aa","Type":"ContainerStarted","Data":"a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0"}
Oct 30 22:12:30.941138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.940954    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:30.941138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.940979    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" event={"ID":"3d8a636e-75a3-4da7-8ccf-058f43de40aa","Type":"ContainerStarted","Data":"49047c76cd06ec5fc4280ba58fd982cdc5c324fed798540d7f178e4ae4ddc07e"}
Oct 30 22:12:30.942551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.942526    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" event={"ID":"2368593a-b34f-4133-b1cd-1e09488a8ca0","Type":"ContainerStarted","Data":"8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd"}
Oct 30 22:12:30.942666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.942566    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" event={"ID":"2368593a-b34f-4133-b1cd-1e09488a8ca0","Type":"ContainerStarted","Data":"40833ef72bf7eb9ff28af42fc38409f0734d06e0bf117a83fe0d44013a491d2d"}
Oct 30 22:12:30.942741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:30.942666    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:31.069547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.069444    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" podStartSLOduration=2.069429661 podStartE2EDuration="2.069429661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:31.068379375 +0000 UTC m=+681.917892349" watchObservedRunningTime="2026-10-30 22:12:31.069429661 +0000 UTC m=+681.918942680"
Oct 30 22:12:31.095553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.095515    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.183554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.183494    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" podStartSLOduration=2.183480871 podStartE2EDuration="2.183480871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:31.182337646 +0000 UTC m=+682.031850625" watchObservedRunningTime="2026-10-30 22:12:31.183480871 +0000 UTC m=+682.032993844"
Oct 30 22:12:31.293980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.293948    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.454922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.454839    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.644610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.644574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.827272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:31.827239    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:34.044834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.044794    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:34.221399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.221363    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:34.435483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.435404    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"]
Oct 30 22:12:34.439449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.439425    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.501712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.501462    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"]
Oct 30 22:12:34.537734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.537675    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxsw\" (UniqueName: \"kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw\") pod \"payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.538074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.538051    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.544971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.544941    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-f80a0e-dockercfg-nl9t7\""
Oct 30 22:12:34.545265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.545245    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-f80a0e\""
Oct 30 22:12:34.562680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.560358    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"]
Oct 30 22:12:34.580708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.565028    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.655716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.649584    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.655716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.649733    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxsw\" (UniqueName: \"kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw\") pod \"payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.659203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.659106    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"]
Oct 30 22:12:34.659203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.659178    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.720917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.720886    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxsw\" (UniqueName: \"kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw\") pod \"payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.754956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.754866    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:34.755182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.755149    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.755419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.755390    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2zn\" (UniqueName: \"kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn\") pod \"payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.857107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.856766    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2zn\" (UniqueName: \"kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn\") pod \"payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.857107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.856858    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.857832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.857790    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.945513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.945442    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2zn\" (UniqueName: \"kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn\") pod \"payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:34.969985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:34.969955    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:35.184325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.184281    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"]
Oct 30 22:12:35.189881 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:35.189834    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58798e6e_4fa9_46e7_8c7c_47aec100691b.slice/crio-750345500b2616e84cde7e63a2435ef7f205dfb23880d24e93f0158fc7862b11 WatchSource:0}: Error finding container 750345500b2616e84cde7e63a2435ef7f205dfb23880d24e93f0158fc7862b11: Status 404 returned error can't find the container with id 750345500b2616e84cde7e63a2435ef7f205dfb23880d24e93f0158fc7862b11
Oct 30 22:12:35.377328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.377289    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"]
Oct 30 22:12:35.382389 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:35.382356    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6e7f7f_4df8_4202_90c6_211da7ad2951.slice/crio-ee85279469d612a2bf96fbc3c0606ffa98924560645a17d2beeaef7bb76a35b1 WatchSource:0}: Error finding container ee85279469d612a2bf96fbc3c0606ffa98924560645a17d2beeaef7bb76a35b1: Status 404 returned error can't find the container with id ee85279469d612a2bf96fbc3c0606ffa98924560645a17d2beeaef7bb76a35b1
Oct 30 22:12:35.975366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.975330    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" event={"ID":"58798e6e-4fa9-46e7-8c7c-47aec100691b","Type":"ContainerStarted","Data":"007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b"}
Oct 30 22:12:35.975556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.975383    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" event={"ID":"58798e6e-4fa9-46e7-8c7c-47aec100691b","Type":"ContainerStarted","Data":"750345500b2616e84cde7e63a2435ef7f205dfb23880d24e93f0158fc7862b11"}
Oct 30 22:12:35.975556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.975424    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:35.976905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.976881    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" event={"ID":"8c6e7f7f-4df8-4202-90c6-211da7ad2951","Type":"ContainerStarted","Data":"f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5"}
Oct 30 22:12:35.977014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.976927    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" event={"ID":"8c6e7f7f-4df8-4202-90c6-211da7ad2951","Type":"ContainerStarted","Data":"ee85279469d612a2bf96fbc3c0606ffa98924560645a17d2beeaef7bb76a35b1"}
Oct 30 22:12:35.977014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:35.977008    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:36.093793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:36.093722    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" podStartSLOduration=2.093703325 podStartE2EDuration="2.093703325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:36.091365885 +0000 UTC m=+686.940878846" watchObservedRunningTime="2026-10-30 22:12:36.093703325 +0000 UTC m=+686.943216296"
Oct 30 22:12:36.206148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:36.206077    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" podStartSLOduration=2.206056037 podStartE2EDuration="2.206056037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:36.204268066 +0000 UTC m=+687.053781041" watchObservedRunningTime="2026-10-30 22:12:36.206056037 +0000 UTC m=+687.055569010"
Oct 30 22:12:36.714865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:36.714835    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.880509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:36.880473    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.945045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:36.943910    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:12:36.946152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:36.946128    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:12:37.038768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:37.027330    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:38.742568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:38.742517    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:38.942259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:38.942217    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:39.306423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.306390    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"]
Oct 30 22:12:39.310242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.310224    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.366035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.366003    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"]
Oct 30 22:12:39.408917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.408889    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-be17e7\""
Oct 30 22:12:39.409071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.408928    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-be17e7-dockercfg-2cnlq\""
Oct 30 22:12:39.505856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.505817    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.506074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.505913    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klpl\" (UniqueName: \"kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl\") pod \"payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.606620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.606518    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klpl\" (UniqueName: \"kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl\") pod \"payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.606794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.606665    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.607507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.607477    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.675499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.675455    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klpl\" (UniqueName: \"kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl\") pod \"payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:39.952645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:39.952557    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:40.325296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:40.325257    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"]
Oct 30 22:12:40.330820 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:12:40.330788    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0f8411_c36e_4301_aaa7_af10dacadb51.slice/crio-a18dec83b4c9db9cfe209e3243bd2889a3c901880231ffdea020948781401cdb WatchSource:0}: Error finding container a18dec83b4c9db9cfe209e3243bd2889a3c901880231ffdea020948781401cdb: Status 404 returned error can't find the container with id a18dec83b4c9db9cfe209e3243bd2889a3c901880231ffdea020948781401cdb
Oct 30 22:12:41.010053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.010013    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" event={"ID":"9d0f8411-c36e-4301-aaa7-af10dacadb51","Type":"ContainerStarted","Data":"37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479"}
Oct 30 22:12:41.010539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.010105    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" event={"ID":"9d0f8411-c36e-4301-aaa7-af10dacadb51","Type":"ContainerStarted","Data":"a18dec83b4c9db9cfe209e3243bd2889a3c901880231ffdea020948781401cdb"}
Oct 30 22:12:41.010539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.010230    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:41.124697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.124614    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" podStartSLOduration=2.124596071 podStartE2EDuration="2.124596071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:41.123636368 +0000 UTC m=+691.973149345" watchObservedRunningTime="2026-10-30 22:12:41.124596071 +0000 UTC m=+691.974109054"
Oct 30 22:12:41.899101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.899052    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:41.958987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.958955    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:12:41.959341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:41.959323    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:12:42.115440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:42.115378    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:42.259658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:42.259627    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:43.403090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.403044    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:43.569544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.569499    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:43.809746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.809712    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"]
Oct 30 22:12:43.812747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.812723    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:43.857007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.856965    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8\") pod \"payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:43.857169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.857097    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:43.911083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.911040    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-eb4aa1\""
Oct 30 22:12:43.911083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.911053    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-eb4aa1-dockercfg-fk7ph\""
Oct 30 22:12:43.914041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.914011    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"]
Oct 30 22:12:43.958020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.957978    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8\") pod \"payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:43.958214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.958087    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:43.958854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:43.958833    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:44.044739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.037383    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8\") pod \"payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:44.045110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.045084    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"]
Oct 30 22:12:44.049158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.049117    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.049921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.049889    5028 generic.go:356] "Generic (PLEG): container finished" podID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerID="32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc" exitCode=0
Oct 30 22:12:44.050051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.049980    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" event={"ID":"e660d4aa-1866-4615-a66b-b5d3d499d00b","Type":"ContainerDied","Data":"32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"}
Oct 30 22:12:44.050560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.050538    5028 scope.go:122] "RemoveContainer" containerID="32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"
Oct 30 22:12:44.052405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.052377    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:44.052516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.052440    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:44.052516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.052475    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:44.090086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.090002    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"]
Oct 30 22:12:44.125766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.125540    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:44.279112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.279073    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.279331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.279311    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6tdj\" (UniqueName: \"kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj\") pod \"payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.393238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.391973    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.393238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.392018    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6tdj\" (UniqueName: \"kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj\") pod \"payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.393478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.393384    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.495893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.495857    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6tdj\" (UniqueName: \"kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj\") pod \"payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.674757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.673955    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:44.863031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:44.862474    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"]
Oct 30 22:12:45.115573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.115536    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" event={"ID":"e660d4aa-1866-4615-a66b-b5d3d499d00b","Type":"ContainerStarted","Data":"e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72"}
Oct 30 22:12:45.135137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.135095    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" event={"ID":"919be555-ac42-475b-932e-fedb50ebea01","Type":"ContainerStarted","Data":"d445804953432f8cd19502e61e1018edf561ae44de7b0e2890e7b5b71f811204"}
Oct 30 22:12:45.206320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.206285    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"]
Oct 30 22:12:45.707052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.706949    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:45.708399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.708363    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:12:45.708523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.708427    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 22:12:45.943414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:45.943378    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:46.101331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.101270    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:46.252718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.246851    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" event={"ID":"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a","Type":"ContainerStarted","Data":"fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54"}
Oct 30 22:12:46.252718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.246922    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" event={"ID":"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a","Type":"ContainerStarted","Data":"8de1d66b3e8c159a85ec2c2004264eebf0c59974b1cb057c8429295476c72820"}
Oct 30 22:12:46.252718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.247833    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:46.276720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.268129    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" event={"ID":"919be555-ac42-475b-932e-fedb50ebea01","Type":"ContainerStarted","Data":"3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924"}
Oct 30 22:12:46.277036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.277007    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:46.468920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.468859    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" podStartSLOduration=3.468834632 podStartE2EDuration="3.468834632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:46.384979252 +0000 UTC m=+697.234492228" watchObservedRunningTime="2026-10-30 22:12:46.468834632 +0000 UTC m=+697.318347603"
Oct 30 22:12:46.500239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.500202    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"]
Oct 30 22:12:46.625710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.562417    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:46.660711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.591870    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" podStartSLOduration=3.591847963 podStartE2EDuration="3.591847963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:46.525896922 +0000 UTC m=+697.375409885" watchObservedRunningTime="2026-10-30 22:12:46.591847963 +0000 UTC m=+697.441360937"
Oct 30 22:12:46.703713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.661483    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqj7\" (UniqueName: \"kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7\") pod \"payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:46.703713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.661573    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:46.713735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.711301    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:12:46.713735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.711357    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 22:12:46.728801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.726978    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d2d4c3-dockercfg-w74km\""
Oct 30 22:12:46.728801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.727711    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d2d4c3\""
Oct 30 22:12:46.806381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.763290    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqj7\" (UniqueName: \"kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7\") pod \"payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:46.806381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.763375    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:46.806381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.764503    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:46.806381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.769794    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"]
Oct 30 22:12:46.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.854611    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:46.896480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:46.896436    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqj7\" (UniqueName: \"kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7\") pod \"payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:47.085767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.085101    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:47.122714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.122644    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:47.134431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.134401    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:12:47.208219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.208186    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:12:47.307563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.307524    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:47.502250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.502168    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:47.708403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.708020    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:12:47.708403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.708069    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 22:12:47.973725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:47.972707    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"]
Oct 30 22:12:48.404026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:48.403939    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" event={"ID":"f5ec1f18-325e-4a7e-b198-8be5c306938d","Type":"ContainerStarted","Data":"3a58f9b69f35ff7b26879846ffed044a67d2acccb1ed0a0a99f3724533d437cd"}
Oct 30 22:12:48.404026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:48.404007    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:48.404026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:48.404030    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" event={"ID":"f5ec1f18-325e-4a7e-b198-8be5c306938d","Type":"ContainerStarted","Data":"a0ade28a5e8bf95e698bbed67731aeec7e31a6797f400c54556f77e1db81780d"}
Oct 30 22:12:48.530815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:48.525838    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" podStartSLOduration=2.525818806 podStartE2EDuration="2.525818806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:48.525103633 +0000 UTC m=+699.374616635" watchObservedRunningTime="2026-10-30 22:12:48.525818806 +0000 UTC m=+699.375331780"
Oct 30 22:12:48.706749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:48.706649    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:12:48.706749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:48.706735    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 22:12:49.706946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:49.706912    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:12:49.707335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:49.706967    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 22:12:50.706782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:50.706740    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:12:50.706948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:50.706805    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 30 22:12:51.711937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:51.711899    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:51.713558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:51.713533    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:12:51.987066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:51.986961    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:52.037645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:52.037616    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:12:54.454017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.453978    5028 generic.go:356] "Generic (PLEG): container finished" podID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerID="9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9" exitCode=0
Oct 30 22:12:54.454512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.454060    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" event={"ID":"cd9a5089-8749-4d9b-b53d-96eb145c757a","Type":"ContainerDied","Data":"9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"}
Oct 30 22:12:54.454512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.454502    5028 scope.go:122] "RemoveContainer" containerID="9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"
Oct 30 22:12:54.456047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.456017    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:54.456155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.456083    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:54.456155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.456101    5028 generic.go:356] "Generic (PLEG): container finished" podID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerID="a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab" exitCode=0
Oct 30 22:12:54.456155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.456124    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:54.456155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.456127    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" event={"ID":"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9","Type":"ContainerDied","Data":"a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"}
Oct 30 22:12:54.456435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.456422    5028 scope.go:122] "RemoveContainer" containerID="a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"
Oct 30 22:12:54.457592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.457569    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:54.457655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.457625    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:54.457708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:54.457658    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:55.486996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:55.486951    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" event={"ID":"cd9a5089-8749-4d9b-b53d-96eb145c757a","Type":"ContainerStarted","Data":"4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320"}
Oct 30 22:12:55.489525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:55.489497    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" event={"ID":"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9","Type":"ContainerStarted","Data":"de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f"}
Oct 30 22:12:55.636727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:55.635803    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:12:55.636727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:55.635880    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:12:55.636727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:55.636169    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:12:56.328709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.327827    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:12:56.328709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.328271    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:12:56.328709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.328394    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:12:56.624769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.621117    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:12:56.624769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.621188    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:12:56.752711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.743243    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:56.897141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:56.897024    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:57.049660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:57.049568    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:57.330143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:57.330110    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.330450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:57.330420    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:12:57.439709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:57.439278    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:12:57.624769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:57.621093    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.624769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:57.621173    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:12:58.323120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:58.323081    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.323402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:58.323371    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:12:58.407346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:58.407315    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:12:58.488671 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:12:58.488634    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:12:58.648767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:58.642004    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.649984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:58.649556    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:12:59.327152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:59.327115    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.327436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:59.327410    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:12:59.415045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:59.415021    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:12:59.657765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:59.637786    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.657765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:12:59.637857    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:13:00.337712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:00.332426    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.337712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:00.332483    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 22:13:00.620153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:00.620053    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.620153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:00.620120    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:13:01.327451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:01.327417    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:13:01.327902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:01.327881    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:13:01.328649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:01.328630    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:13:01.625524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:01.625334    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:13:01.625757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:01.625733    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:13:01.626481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:01.626457    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:13:02.070789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:02.070756    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:03.136722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:03.136672    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:06.155059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:06.155023    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:06.893682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:06.893647    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:09.967265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:09.967227    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:10.614823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:10.612462    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:10.818442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:10.818397    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.593741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:12.593705    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.775374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:12.775335    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.937275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:12.937181    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:15.189179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:15.189143    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:15.642722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:15.642673    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:16.266826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:16.266780    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:16.418791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:16.418759    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:16.571439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:16.571341    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:18.016582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:18.016541    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:22.986267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:22.986209    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.375163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:28.375125    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.464256 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:13:28.464215    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:13:28.531805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:28.531754    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:29.042952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:29.042904    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:29.225866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:29.225826    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:29.392882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:29.392791    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:31.349012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:31.348969    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.080513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.080455    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.232086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.232003    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.783408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.783372    5028 generic.go:356] "Generic (PLEG): container finished" podID="b242cced-73d5-48cf-9819-84803277f9da" containerID="a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908" exitCode=0
Oct 30 22:13:32.783932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.783452    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" event={"ID":"b242cced-73d5-48cf-9819-84803277f9da","Type":"ContainerDied","Data":"a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"}
Oct 30 22:13:32.784000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.783962    5028 scope.go:122] "RemoveContainer" containerID="a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"
Oct 30 22:13:32.785354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.785325    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:32.785446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.785390    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:32.785446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:32.785422    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:33.628732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:33.628673    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:33.791503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:33.791466    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" event={"ID":"b242cced-73d5-48cf-9819-84803277f9da","Type":"ContainerStarted","Data":"f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc"}
Oct 30 22:13:33.862530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:33.862496    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:13:34.540513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:34.540479    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:13:34.542857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:34.542826    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:13:34.543149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:34.543105    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:13:35.541713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:35.541159    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:13:35.541713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:35.541236    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:13:36.542133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:36.542093    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:13:36.542584 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:36.542163    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:13:36.650334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:36.650302    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.326088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:37.326051    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.488848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:37.488811    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:37.541857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:37.541816    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:13:37.542066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:37.541868    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:13:38.542108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:38.542074    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:13:38.542514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:38.542130    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 22:13:38.881389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:38.881303    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:39.544060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:39.544030    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:13:39.545091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:39.545074    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:13:41.752840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:41.752807    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:41.897892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:41.897860    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:43.662901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:43.662869    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:45.967160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:45.967124    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:46.624070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:46.624018    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:46.809490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:46.809460    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.001752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:47.001713    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:47.204575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:47.204541    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:49.002134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:49.002100    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:51.272991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:51.272956    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:53.889093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:53.889061    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:58.435386 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:13:58.435351    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:13:58.782835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:13:58.782798    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:03.010591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:03.010556    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:03.469655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:03.469602    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:03.651817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:03.651780    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:03.803724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:03.803629    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:04.070147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:04.070061    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:04.226436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:04.226400    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:04.615280 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:04.615244    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:08.893927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:08.893872    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:12.363333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:12.363286    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:13.776977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:13.776943    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:18.801778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:18.801738    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.138466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:22.138431    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.653111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:23.653074    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:26.129711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:26.129653    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:28.436023 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:14:28.435981    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:14:28.518522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:28.518473    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:33.315901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:33.315859    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:33.885338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:33.885302    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:34.030896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:34.030849    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:35.296906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.296868    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"]
Oct 30 22:14:35.299870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.299848    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.351333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.351294    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"openshift-service-ca.crt\""
Oct 30 22:14:35.398645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.398610    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"kube-root-ca.crt\""
Oct 30 22:14:35.398858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.398729    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"simulator-w0-d2d4c3-kserve-self-signed-certs\""
Oct 30 22:14:35.398858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.398817    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"default-dockercfg-6l7bq\""
Oct 30 22:14:35.400133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.400109    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:35.402739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.402715    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"]
Oct 30 22:14:35.428922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.428887    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.429041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.428983    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.429041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.429033    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n48nc\" (UniqueName: \"kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.429135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.429059    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.429295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.429273    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.429408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.429385    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.530653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.530616    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.530880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.530741    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.530880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.530838    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.530880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.530874    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.531068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.530913    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.531068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.530955    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n48nc\" (UniqueName: \"kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.531521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.531488    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.531650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.531533    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.531795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.531773    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.533593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.533566    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.533733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.533677    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.589956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.589883    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n48nc\" (UniqueName: \"kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc\") pod \"simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") " pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.610599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.610566    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:35.990164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:35.990136    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"]
Oct 30 22:14:35.990916 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:14:35.990891    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd239f8_de79_43a0_af71_bf140440a576.slice/crio-0cde920eabdff9481daf69a25c1fb4587a2f1f7ce935d08850e4d079a9fd43f0 WatchSource:0}: Error finding container 0cde920eabdff9481daf69a25c1fb4587a2f1f7ce935d08850e4d079a9fd43f0: Status 404 returned error can't find the container with id 0cde920eabdff9481daf69a25c1fb4587a2f1f7ce935d08850e4d079a9fd43f0
Oct 30 22:14:36.185244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:36.185204    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" event={"ID":"ccd239f8-de79-43a0-af71-bf140440a576","Type":"ContainerStarted","Data":"dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087"}
Oct 30 22:14:36.185412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:36.185256    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" event={"ID":"ccd239f8-de79-43a0-af71-bf140440a576","Type":"ContainerStarted","Data":"0cde920eabdff9481daf69a25c1fb4587a2f1f7ce935d08850e4d079a9fd43f0"}
Oct 30 22:14:36.303174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:36.303062    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podStartSLOduration=1.303049175 podStartE2EDuration="1.303049175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:36.300004126 +0000 UTC m=+807.149517100" watchObservedRunningTime="2026-10-30 22:14:36.303049175 +0000 UTC m=+807.152562148"
Oct 30 22:14:36.611567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:36.611488    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:38.162189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:38.162155    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:38.486640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:38.486601    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:38.666782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:38.666743    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:42.957839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:42.957807    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:45.611090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:45.611051    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:45.623834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:45.623805    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:46.263551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:46.263521    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:14:47.897234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:47.897199    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:52.885645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:52.885608    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:57.760036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:14:57.760000    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:58.436421 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:14:58.436382    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:15:00.188421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.188387    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"]
Oct 30 22:15:00.191423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.191394    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"]
Oct 30 22:15:00.191606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.191494    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.208419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.208388    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"]
Oct 30 22:15:00.208556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.208509    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.211512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.211491    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.235325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.235294    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"]
Oct 30 22:15:00.238430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.238407    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"]
Oct 30 22:15:00.238551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.238515    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.242035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.242011    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.243282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.243265    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:00.243282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.243278    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:00.243414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.243333    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:15:00.283381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.283344    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"]
Oct 30 22:15:00.284264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.284227    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.284413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.284295    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzhc\" (UniqueName: \"kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.284413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.284348    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.284530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.284466    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvc2g\" (UniqueName: \"kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.284530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.284501    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.284703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.284658    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6l7q\" (UniqueName: \"kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.286292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.286272    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"]
Oct 30 22:15:00.288649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.288625    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"]
Oct 30 22:15:00.321384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.321341    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"]
Oct 30 22:15:00.325222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.325192    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"]
Oct 30 22:15:00.386053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386019    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzhc\" (UniqueName: \"kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.386053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386055    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.386347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386306    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.386422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386383    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5mw\" (UniqueName: \"kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.386490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386452    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvc2g\" (UniqueName: \"kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.386490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386480    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.386751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386732    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whh4d\" (UniqueName: \"kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.386860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386845    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6l7q\" (UniqueName: \"kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.386986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386909    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.386986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386923    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.386986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.386937    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.387234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.387217    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.387555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.387538    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.445649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.445581    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzhc\" (UniqueName: \"kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.445866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.445842    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvc2g\" (UniqueName: \"kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.445917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.445863    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6l7q\" (UniqueName: \"kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.488297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.488260    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.488470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.488354    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.488470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.488379    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5mw\" (UniqueName: \"kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.488470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.488451    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whh4d\" (UniqueName: \"kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.489023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.488996    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.489163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.489088    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.510621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.510594    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:00.521616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.521586    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:00.529076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.529047    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:00.550735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.548941    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whh4d\" (UniqueName: \"kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.550735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.549596    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5mw\" (UniqueName: \"kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.552360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.552320    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:00.561718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.561675    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:00.900166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.900130    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"]
Oct 30 22:15:00.900743 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:00.900715    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde188dba_7ec0_4328_b09c_2f3b9dab348d.slice/crio-6b8b61af2e5bcbfc42a1e880258c7b96782a1bc404bd062bf948c34644a014d0 WatchSource:0}: Error finding container 6b8b61af2e5bcbfc42a1e880258c7b96782a1bc404bd062bf948c34644a014d0: Status 404 returned error can't find the container with id 6b8b61af2e5bcbfc42a1e880258c7b96782a1bc404bd062bf948c34644a014d0
Oct 30 22:15:00.947617 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:00.947586    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc556a798_9568_429a_b952_c380629cc322.slice/crio-cc547d54e0ebdc941d51012799a8300129462e309f24033aefa3f515cd7cce8b WatchSource:0}: Error finding container cc547d54e0ebdc941d51012799a8300129462e309f24033aefa3f515cd7cce8b: Status 404 returned error can't find the container with id cc547d54e0ebdc941d51012799a8300129462e309f24033aefa3f515cd7cce8b
Oct 30 22:15:00.947855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.947830    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"]
Oct 30 22:15:00.951084 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:00.951062    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422cdcf0_bced_4e1e_89d0_b475f6e86b7d.slice/crio-7602335d6ff99b5422ec77e1f33122306da0e61e60a31bf261e396e77f755de2 WatchSource:0}: Error finding container 7602335d6ff99b5422ec77e1f33122306da0e61e60a31bf261e396e77f755de2: Status 404 returned error can't find the container with id 7602335d6ff99b5422ec77e1f33122306da0e61e60a31bf261e396e77f755de2
Oct 30 22:15:00.951418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.951390    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"]
Oct 30 22:15:00.997140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:00.997109    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"]
Oct 30 22:15:00.998000 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:00.997972    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a1c655_faa9_4885_b528_9f33bf75f7a6.slice/crio-f2d45a6c462058c508071b3f7b10b7baf4c719f95afa72cb41c5c8ce097aebad WatchSource:0}: Error finding container f2d45a6c462058c508071b3f7b10b7baf4c719f95afa72cb41c5c8ce097aebad: Status 404 returned error can't find the container with id f2d45a6c462058c508071b3f7b10b7baf4c719f95afa72cb41c5c8ce097aebad
Oct 30 22:15:01.000238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:01.000211    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"]
Oct 30 22:15:01.000922 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:01.000891    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d33543_6a75_4d8c_ac47_a5674327f173.slice/crio-170a0a7c26801414f88a07a82db6901e4d0508f5943b0dc0ec27f1b183926221 WatchSource:0}: Error finding container 170a0a7c26801414f88a07a82db6901e4d0508f5943b0dc0ec27f1b183926221: Status 404 returned error can't find the container with id 170a0a7c26801414f88a07a82db6901e4d0508f5943b0dc0ec27f1b183926221
Oct 30 22:15:01.347703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:01.347658    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerStarted","Data":"f2d45a6c462058c508071b3f7b10b7baf4c719f95afa72cb41c5c8ce097aebad"}
Oct 30 22:15:01.349175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:01.349143    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" event={"ID":"c556a798-9568-429a-b952-c380629cc322","Type":"ContainerStarted","Data":"cc547d54e0ebdc941d51012799a8300129462e309f24033aefa3f515cd7cce8b"}
Oct 30 22:15:01.351229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:01.351204    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" event={"ID":"53d33543-6a75-4d8c-ac47-a5674327f173","Type":"ContainerStarted","Data":"170a0a7c26801414f88a07a82db6901e4d0508f5943b0dc0ec27f1b183926221"}
Oct 30 22:15:01.352509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:01.352481    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" event={"ID":"de188dba-7ec0-4328-b09c-2f3b9dab348d","Type":"ContainerStarted","Data":"6b8b61af2e5bcbfc42a1e880258c7b96782a1bc404bd062bf948c34644a014d0"}
Oct 30 22:15:01.353880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:01.353859    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerStarted","Data":"7602335d6ff99b5422ec77e1f33122306da0e61e60a31bf261e396e77f755de2"}
Oct 30 22:15:02.589778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:02.589744    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:03.816754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:03.816707    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:05.475187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:05.475153    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:07.443385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:07.443311    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" event={"ID":"53d33543-6a75-4d8c-ac47-a5674327f173","Type":"ContainerStarted","Data":"4900305f3346931fea3bcb489e7647b867367b35e36b71ceb95b2b02baed2c4e"}
Oct 30 22:15:07.564061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:07.563841    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" podStartSLOduration=2.286741039 podStartE2EDuration="7.563819137s" totalImagesPullingTime="5.277078098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.555476517 +0000 UTC m=+838.404989483" watchObservedRunningTime="2026-10-30 22:15:07.563819137 +0000 UTC m=+838.413332111"
Oct 30 22:15:07.577273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:07.577238    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:08.227413 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:08.227380    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc556a798_9568_429a_b952_c380629cc322.slice/crio-f2854c63f39a8b5246091602552b420e7c27c7894c4f655a629ef04bc70a1558.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:15:08.452709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.452600    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerStarted","Data":"d0d71fc77f256b23cc9dbd6b44ddaf344dce2e2338e33bcda339b9a529f7d17c"}
Oct 30 22:15:08.454370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.454342    5028 generic.go:356] "Generic (PLEG): container finished" podID="c556a798-9568-429a-b952-c380629cc322" containerID="f2854c63f39a8b5246091602552b420e7c27c7894c4f655a629ef04bc70a1558" exitCode=60
Oct 30 22:15:08.454507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.454405    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" event={"ID":"c556a798-9568-429a-b952-c380629cc322","Type":"ContainerDied","Data":"f2854c63f39a8b5246091602552b420e7c27c7894c4f655a629ef04bc70a1558"}
Oct 30 22:15:08.454672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.454657    5028 scope.go:122] "RemoveContainer" containerID="f2854c63f39a8b5246091602552b420e7c27c7894c4f655a629ef04bc70a1558"
Oct 30 22:15:08.456321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.456295    5028 generic.go:356] "Generic (PLEG): container finished" podID="53d33543-6a75-4d8c-ac47-a5674327f173" containerID="4900305f3346931fea3bcb489e7647b867367b35e36b71ceb95b2b02baed2c4e" exitCode=60
Oct 30 22:15:08.456435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.456400    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" event={"ID":"53d33543-6a75-4d8c-ac47-a5674327f173","Type":"ContainerDied","Data":"4900305f3346931fea3bcb489e7647b867367b35e36b71ceb95b2b02baed2c4e"}
Oct 30 22:15:08.456625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.456607    5028 scope.go:122] "RemoveContainer" containerID="4900305f3346931fea3bcb489e7647b867367b35e36b71ceb95b2b02baed2c4e"
Oct 30 22:15:08.458735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.458623    5028 generic.go:356] "Generic (PLEG): container finished" podID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerID="da2d4a0d42b1ec4f66724a0c095644bcfe56656c3a1dd8bf2f136022282e7668" exitCode=60
Oct 30 22:15:08.458735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.458726    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" event={"ID":"de188dba-7ec0-4328-b09c-2f3b9dab348d","Type":"ContainerDied","Data":"da2d4a0d42b1ec4f66724a0c095644bcfe56656c3a1dd8bf2f136022282e7668"}
Oct 30 22:15:08.458977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.458965    5028 scope.go:122] "RemoveContainer" containerID="da2d4a0d42b1ec4f66724a0c095644bcfe56656c3a1dd8bf2f136022282e7668"
Oct 30 22:15:08.461497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.461465    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerStarted","Data":"bea06863e6d3d037340d121b95dd75b87ce612d3aca530e58c8f4c0042b1cb2b"}
Oct 30 22:15:08.570336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:08.570280    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" podStartSLOduration=1.905360196 podStartE2EDuration="8.570265445s" totalImagesPullingTime="6.664905249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:08.567752132 +0000 UTC m=+839.417265108" watchObservedRunningTime="2026-10-30 22:15:08.570265445 +0000 UTC m=+839.419778418"
Oct 30 22:15:09.019039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.018919    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" podStartSLOduration=2.3020550650000002 podStartE2EDuration="9.018905514s" totalImagesPullingTime="6.716850449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:09.01795356 +0000 UTC m=+839.867466561" watchObservedRunningTime="2026-10-30 22:15:09.018905514 +0000 UTC m=+839.868418486"
Oct 30 22:15:09.477442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.477405    5028 generic.go:356] "Generic (PLEG): container finished" podID="c556a798-9568-429a-b952-c380629cc322" containerID="c436dd71ef0ba180f852a96f56410a38c159475d7145668f7f7a67f0a1a62e55" exitCode=60
Oct 30 22:15:09.477904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.477492    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" event={"ID":"c556a798-9568-429a-b952-c380629cc322","Type":"ContainerDied","Data":"c436dd71ef0ba180f852a96f56410a38c159475d7145668f7f7a67f0a1a62e55"}
Oct 30 22:15:09.477904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.477622    5028 scope.go:122] "RemoveContainer" containerID="f2854c63f39a8b5246091602552b420e7c27c7894c4f655a629ef04bc70a1558"
Oct 30 22:15:09.477904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.477785    5028 scope.go:122] "RemoveContainer" containerID="c436dd71ef0ba180f852a96f56410a38c159475d7145668f7f7a67f0a1a62e55"
Oct 30 22:15:09.478093 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:09.478069    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg_odh-ai-gateway-infra(c556a798-9568-429a-b952-c380629cc322)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" podUID="c556a798-9568-429a-b952-c380629cc322"
Oct 30 22:15:09.479518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.479497    5028 generic.go:356] "Generic (PLEG): container finished" podID="53d33543-6a75-4d8c-ac47-a5674327f173" containerID="1312bd9992cdb7deaf3cd7ed64e919ef7fe6a2e61c6beb0d3c30993b03fd9e44" exitCode=60
Oct 30 22:15:09.479624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.479580    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" event={"ID":"53d33543-6a75-4d8c-ac47-a5674327f173","Type":"ContainerDied","Data":"1312bd9992cdb7deaf3cd7ed64e919ef7fe6a2e61c6beb0d3c30993b03fd9e44"}
Oct 30 22:15:09.479814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.479799    5028 scope.go:122] "RemoveContainer" containerID="1312bd9992cdb7deaf3cd7ed64e919ef7fe6a2e61c6beb0d3c30993b03fd9e44"
Oct 30 22:15:09.480048 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:09.480022    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2_odh-ai-gateway-infra(53d33543-6a75-4d8c-ac47-a5674327f173)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" podUID="53d33543-6a75-4d8c-ac47-a5674327f173"
Oct 30 22:15:09.481579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.481556    5028 generic.go:356] "Generic (PLEG): container finished" podID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerID="9b80dff8f38ef314899171d560d8fa02ee3adb054bb18149c4a9dc27aaade26e" exitCode=60
Oct 30 22:15:09.481670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.481630    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" event={"ID":"de188dba-7ec0-4328-b09c-2f3b9dab348d","Type":"ContainerDied","Data":"9b80dff8f38ef314899171d560d8fa02ee3adb054bb18149c4a9dc27aaade26e"}
Oct 30 22:15:09.481924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.481904    5028 scope.go:122] "RemoveContainer" containerID="9b80dff8f38ef314899171d560d8fa02ee3adb054bb18149c4a9dc27aaade26e"
Oct 30 22:15:09.482190 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:09.482156    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw_odh-ai-gateway-infra(de188dba-7ec0-4328-b09c-2f3b9dab348d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d"
Oct 30 22:15:09.483524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.483502    5028 generic.go:356] "Generic (PLEG): container finished" podID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerID="bea06863e6d3d037340d121b95dd75b87ce612d3aca530e58c8f4c0042b1cb2b" exitCode=60
Oct 30 22:15:09.483662 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.483635    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerDied","Data":"bea06863e6d3d037340d121b95dd75b87ce612d3aca530e58c8f4c0042b1cb2b"}
Oct 30 22:15:09.483853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.483839    5028 scope.go:122] "RemoveContainer" containerID="bea06863e6d3d037340d121b95dd75b87ce612d3aca530e58c8f4c0042b1cb2b"
Oct 30 22:15:09.485421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.485400    5028 generic.go:356] "Generic (PLEG): container finished" podID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerID="d0d71fc77f256b23cc9dbd6b44ddaf344dce2e2338e33bcda339b9a529f7d17c" exitCode=60
Oct 30 22:15:09.485477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.485454    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerDied","Data":"d0d71fc77f256b23cc9dbd6b44ddaf344dce2e2338e33bcda339b9a529f7d17c"}
Oct 30 22:15:09.485604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.485592    5028 scope.go:122] "RemoveContainer" containerID="d0d71fc77f256b23cc9dbd6b44ddaf344dce2e2338e33bcda339b9a529f7d17c"
Oct 30 22:15:09.593817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.593794    5028 scope.go:122] "RemoveContainer" containerID="4900305f3346931fea3bcb489e7647b867367b35e36b71ceb95b2b02baed2c4e"
Oct 30 22:15:09.612330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:09.612307    5028 scope.go:122] "RemoveContainer" containerID="da2d4a0d42b1ec4f66724a0c095644bcfe56656c3a1dd8bf2f136022282e7668"
Oct 30 22:15:10.493455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:10.493427    5028 scope.go:122] "RemoveContainer" containerID="9b80dff8f38ef314899171d560d8fa02ee3adb054bb18149c4a9dc27aaade26e"
Oct 30 22:15:10.493943 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:10.493777    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw_odh-ai-gateway-infra(de188dba-7ec0-4328-b09c-2f3b9dab348d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d"
Oct 30 22:15:10.495465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:10.495437    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerStarted","Data":"96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44"}
Oct 30 22:15:10.497544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:10.497514    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerStarted","Data":"2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d"}
Oct 30 22:15:10.499219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:10.499199    5028 scope.go:122] "RemoveContainer" containerID="c436dd71ef0ba180f852a96f56410a38c159475d7145668f7f7a67f0a1a62e55"
Oct 30 22:15:10.499437 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:10.499415    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg_odh-ai-gateway-infra(c556a798-9568-429a-b952-c380629cc322)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" podUID="c556a798-9568-429a-b952-c380629cc322"
Oct 30 22:15:10.500735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:10.500718    5028 scope.go:122] "RemoveContainer" containerID="1312bd9992cdb7deaf3cd7ed64e919ef7fe6a2e61c6beb0d3c30993b03fd9e44"
Oct 30 22:15:10.500932 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:10.500912    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2_odh-ai-gateway-infra(53d33543-6a75-4d8c-ac47-a5674327f173)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" podUID="53d33543-6a75-4d8c-ac47-a5674327f173"
Oct 30 22:15:11.508993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.508958    5028 generic.go:356] "Generic (PLEG): container finished" podID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerID="96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44" exitCode=60
Oct 30 22:15:11.509448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.509042    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerDied","Data":"96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44"}
Oct 30 22:15:11.509448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.509101    5028 scope.go:122] "RemoveContainer" containerID="bea06863e6d3d037340d121b95dd75b87ce612d3aca530e58c8f4c0042b1cb2b"
Oct 30 22:15:11.509448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.509303    5028 scope.go:122] "RemoveContainer" containerID="96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44"
Oct 30 22:15:11.509612 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:11.509587    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7_odh-ai-gateway-infra(422cdcf0-bced-4e1e-89d0-b475f6e86b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d"
Oct 30 22:15:11.511237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.511213    5028 generic.go:356] "Generic (PLEG): container finished" podID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerID="2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d" exitCode=60
Oct 30 22:15:11.511334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.511294    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerDied","Data":"2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d"}
Oct 30 22:15:11.511521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.511509    5028 scope.go:122] "RemoveContainer" containerID="2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d"
Oct 30 22:15:11.511802 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:11.511784    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw_odh-ai-gateway-infra(98a1c655-faa9-4885-b528-9f33bf75f7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6"
Oct 30 22:15:11.520790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:11.520665    5028 scope.go:122] "RemoveContainer" containerID="d0d71fc77f256b23cc9dbd6b44ddaf344dce2e2338e33bcda339b9a529f7d17c"
Oct 30 22:15:12.439264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:12.439227    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:12.519702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:12.519661    5028 scope.go:122] "RemoveContainer" containerID="96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44"
Oct 30 22:15:12.520134 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:12.519985    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7_odh-ai-gateway-infra(422cdcf0-bced-4e1e-89d0-b475f6e86b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d"
Oct 30 22:15:12.521524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:12.521506    5028 scope.go:122] "RemoveContainer" containerID="2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d"
Oct 30 22:15:12.521777 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:12.521757    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw_odh-ai-gateway-infra(98a1c655-faa9-4885-b528-9f33bf75f7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6"
Oct 30 22:15:17.289963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:17.289923    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:18.488856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:18.488822    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:20.284102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:20.284030    5028 scope.go:122] "RemoveContainer" containerID="1312bd9992cdb7deaf3cd7ed64e919ef7fe6a2e61c6beb0d3c30993b03fd9e44"
Oct 30 22:15:21.283988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:21.283962    5028 scope.go:122] "RemoveContainer" containerID="2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d"
Oct 30 22:15:21.591901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:21.591871    5028 generic.go:356] "Generic (PLEG): container finished" podID="53d33543-6a75-4d8c-ac47-a5674327f173" containerID="61739b552cb9f1d23c646992899350aaa9ec5b0bed3582850e3e2ba85668cb95" exitCode=60
Oct 30 22:15:21.592300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:21.591951    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" event={"ID":"53d33543-6a75-4d8c-ac47-a5674327f173","Type":"ContainerDied","Data":"61739b552cb9f1d23c646992899350aaa9ec5b0bed3582850e3e2ba85668cb95"}
Oct 30 22:15:21.592300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:21.592017    5028 scope.go:122] "RemoveContainer" containerID="1312bd9992cdb7deaf3cd7ed64e919ef7fe6a2e61c6beb0d3c30993b03fd9e44"
Oct 30 22:15:21.592381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:21.592369    5028 scope.go:122] "RemoveContainer" containerID="61739b552cb9f1d23c646992899350aaa9ec5b0bed3582850e3e2ba85668cb95"
Oct 30 22:15:21.592663 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:21.592639    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2_odh-ai-gateway-infra(53d33543-6a75-4d8c-ac47-a5674327f173)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" podUID="53d33543-6a75-4d8c-ac47-a5674327f173"
Oct 30 22:15:22.200128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.200096    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:22.601447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.601412    5028 generic.go:356] "Generic (PLEG): container finished" podID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerID="f4532f69c8fb7eaaa2e15445833635bd4f23ce9148fca3fd0ac93716be53ea2d" exitCode=60
Oct 30 22:15:22.601908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.601493    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerDied","Data":"f4532f69c8fb7eaaa2e15445833635bd4f23ce9148fca3fd0ac93716be53ea2d"}
Oct 30 22:15:22.601908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.601569    5028 scope.go:122] "RemoveContainer" containerID="2f292dfb515d316f9100360e5a07edac7361319aa5a24e389cff13d83f9fc83d"
Oct 30 22:15:22.602024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.601960    5028 scope.go:122] "RemoveContainer" containerID="f4532f69c8fb7eaaa2e15445833635bd4f23ce9148fca3fd0ac93716be53ea2d"
Oct 30 22:15:22.602279 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:22.602256    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw_odh-ai-gateway-infra(98a1c655-faa9-4885-b528-9f33bf75f7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6"
Oct 30 22:15:22.717078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.717042    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"]
Oct 30 22:15:22.865970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.865903    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:22.997872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.997821    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d\" (UniqueName: \"kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d\") pod \"53d33543-6a75-4d8c-ac47-a5674327f173\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") "
Oct 30 22:15:22.998072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.997973    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca\") pod \"53d33543-6a75-4d8c-ac47-a5674327f173\" (UID: \"53d33543-6a75-4d8c-ac47-a5674327f173\") "
Oct 30 22:15:22.998475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:22.998447    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca" pod "53d33543-6a75-4d8c-ac47-a5674327f173" (UID: "53d33543-6a75-4d8c-ac47-a5674327f173"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:23.000400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.000352    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d" pod "53d33543-6a75-4d8c-ac47-a5674327f173" (UID: "53d33543-6a75-4d8c-ac47-a5674327f173"). InnerVolumeSpecName "kube-api-access-whh4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:23.099240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.099201    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whh4d\" (UniqueName: \"kubernetes.io/projected/53d33543-6a75-4d8c-ac47-a5674327f173-kube-api-access-whh4d\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.099240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.099233    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d33543-6a75-4d8c-ac47-a5674327f173-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.287299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.287257    5028 scope.go:122] "RemoveContainer" containerID="9b80dff8f38ef314899171d560d8fa02ee3adb054bb18149c4a9dc27aaade26e"
Oct 30 22:15:23.288099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.288075    5028 scope.go:122] "RemoveContainer" containerID="96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44"
Oct 30 22:15:23.614651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.614565    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"
Oct 30 22:15:23.614651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.614602    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2" event={"ID":"53d33543-6a75-4d8c-ac47-a5674327f173","Type":"ContainerDied","Data":"170a0a7c26801414f88a07a82db6901e4d0508f5943b0dc0ec27f1b183926221"}
Oct 30 22:15:23.615171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.614665    5028 scope.go:122] "RemoveContainer" containerID="61739b552cb9f1d23c646992899350aaa9ec5b0bed3582850e3e2ba85668cb95"
Oct 30 22:15:23.732291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.732258    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"]
Oct 30 22:15:23.777823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.777788    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"]
Oct 30 22:15:23.784355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.784325    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889975-t4mf2"]
Oct 30 22:15:23.886628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:23.886603    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:24.008573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.008537    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca\") pod \"98a1c655-faa9-4885-b528-9f33bf75f7a6\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") "
Oct 30 22:15:24.008812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.008791    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw\" (UniqueName: \"kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw\") pod \"98a1c655-faa9-4885-b528-9f33bf75f7a6\" (UID: \"98a1c655-faa9-4885-b528-9f33bf75f7a6\") "
Oct 30 22:15:24.009070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.009030    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca" pod "98a1c655-faa9-4885-b528-9f33bf75f7a6" (UID: "98a1c655-faa9-4885-b528-9f33bf75f7a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:24.009387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.009210    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1c655-faa9-4885-b528-9f33bf75f7a6-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:24.012048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.012018    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw" pod "98a1c655-faa9-4885-b528-9f33bf75f7a6" (UID: "98a1c655-faa9-4885-b528-9f33bf75f7a6"). InnerVolumeSpecName "kube-api-access-ml5mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:24.110706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.110652    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5mw\" (UniqueName: \"kubernetes.io/projected/98a1c655-faa9-4885-b528-9f33bf75f7a6-kube-api-access-ml5mw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:24.167503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.167393    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:24.284238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.283851    5028 scope.go:122] "RemoveContainer" containerID="c436dd71ef0ba180f852a96f56410a38c159475d7145668f7f7a67f0a1a62e55"
Oct 30 22:15:24.625530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.625497    5028 generic.go:356] "Generic (PLEG): container finished" podID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerID="3910a69b8c02a07d0cbcad0f9fe5ef7c099356464732b30d95b84777de87f358" exitCode=60
Oct 30 22:15:24.625999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.625572    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" event={"ID":"de188dba-7ec0-4328-b09c-2f3b9dab348d","Type":"ContainerDied","Data":"3910a69b8c02a07d0cbcad0f9fe5ef7c099356464732b30d95b84777de87f358"}
Oct 30 22:15:24.625999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.625633    5028 scope.go:122] "RemoveContainer" containerID="9b80dff8f38ef314899171d560d8fa02ee3adb054bb18149c4a9dc27aaade26e"
Oct 30 22:15:24.626121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.626091    5028 scope.go:122] "RemoveContainer" containerID="3910a69b8c02a07d0cbcad0f9fe5ef7c099356464732b30d95b84777de87f358"
Oct 30 22:15:24.626628 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:24.626406    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw_odh-ai-gateway-infra(de188dba-7ec0-4328-b09c-2f3b9dab348d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d"
Oct 30 22:15:24.628104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.628076    5028 generic.go:356] "Generic (PLEG): container finished" podID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerID="93f3d2f8676014f1c32eb9aa4d4c262ec64a9962352045aedbd0698ca342bb31" exitCode=60
Oct 30 22:15:24.628271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.628250    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerDied","Data":"93f3d2f8676014f1c32eb9aa4d4c262ec64a9962352045aedbd0698ca342bb31"}
Oct 30 22:15:24.628663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.628641    5028 scope.go:122] "RemoveContainer" containerID="93f3d2f8676014f1c32eb9aa4d4c262ec64a9962352045aedbd0698ca342bb31"
Oct 30 22:15:24.628962 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:24.628939    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7_odh-ai-gateway-infra(422cdcf0-bced-4e1e-89d0-b475f6e86b7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d"
Oct 30 22:15:24.630390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.630373    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"
Oct 30 22:15:24.630753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.630724    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw" event={"ID":"98a1c655-faa9-4885-b528-9f33bf75f7a6","Type":"ContainerDied","Data":"f2d45a6c462058c508071b3f7b10b7baf4c719f95afa72cb41c5c8ce097aebad"}
Oct 30 22:15:24.640169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.640105    5028 scope.go:122] "RemoveContainer" containerID="96a36b0d43c40c7480acaccfef319590d69f2ab2bb70bb2edb212d0356134b44"
Oct 30 22:15:24.653675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.653646    5028 scope.go:122] "RemoveContainer" containerID="f4532f69c8fb7eaaa2e15445833635bd4f23ce9148fca3fd0ac93716be53ea2d"
Oct 30 22:15:24.850357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.850277    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"]
Oct 30 22:15:24.854730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:24.854682    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889975-vg8hw"]
Oct 30 22:15:25.292243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.292208    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" path="/var/lib/kubelet/pods/53d33543-6a75-4d8c-ac47-a5674327f173/volumes"
Oct 30 22:15:25.292631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.292618    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" path="/var/lib/kubelet/pods/98a1c655-faa9-4885-b528-9f33bf75f7a6/volumes"
Oct 30 22:15:25.640564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.640521    5028 generic.go:356] "Generic (PLEG): container finished" podID="c556a798-9568-429a-b952-c380629cc322" containerID="5389e85001657880407ac4ed078ee900c2abba27e84f33480f9f4077bd4b677c" exitCode=60
Oct 30 22:15:25.641081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.640679    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" event={"ID":"c556a798-9568-429a-b952-c380629cc322","Type":"ContainerDied","Data":"5389e85001657880407ac4ed078ee900c2abba27e84f33480f9f4077bd4b677c"}
Oct 30 22:15:25.641081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.640765    5028 scope.go:122] "RemoveContainer" containerID="c436dd71ef0ba180f852a96f56410a38c159475d7145668f7f7a67f0a1a62e55"
Oct 30 22:15:25.641273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.641140    5028 scope.go:122] "RemoveContainer" containerID="5389e85001657880407ac4ed078ee900c2abba27e84f33480f9f4077bd4b677c"
Oct 30 22:15:25.641458 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:25.641434    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg_odh-ai-gateway-infra(c556a798-9568-429a-b952-c380629cc322)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" podUID="c556a798-9568-429a-b952-c380629cc322"
Oct 30 22:15:25.749179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.749138    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"]
Oct 30 22:15:25.903587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.903524    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:25.972374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:25.972341    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"]
Oct 30 22:15:26.030921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.030836    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc\" (UniqueName: \"kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc\") pod \"de188dba-7ec0-4328-b09c-2f3b9dab348d\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") "
Oct 30 22:15:26.031096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.030932    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca\") pod \"de188dba-7ec0-4328-b09c-2f3b9dab348d\" (UID: \"de188dba-7ec0-4328-b09c-2f3b9dab348d\") "
Oct 30 22:15:26.031772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.031724    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca" pod "de188dba-7ec0-4328-b09c-2f3b9dab348d" (UID: "de188dba-7ec0-4328-b09c-2f3b9dab348d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:26.035107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.035075    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc" pod "de188dba-7ec0-4328-b09c-2f3b9dab348d" (UID: "de188dba-7ec0-4328-b09c-2f3b9dab348d"). InnerVolumeSpecName "kube-api-access-kzzhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:26.132721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.132654    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de188dba-7ec0-4328-b09c-2f3b9dab348d-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.132815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.132743    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzhc\" (UniqueName: \"kubernetes.io/projected/de188dba-7ec0-4328-b09c-2f3b9dab348d-kube-api-access-kzzhc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.160245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.160168    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:26.233312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.233254    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca\") pod \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") "
Oct 30 22:15:26.233533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.233461    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q\" (UniqueName: \"kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q\") pod \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\" (UID: \"422cdcf0-bced-4e1e-89d0-b475f6e86b7d\") "
Oct 30 22:15:26.233650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.233626    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca" pod "422cdcf0-bced-4e1e-89d0-b475f6e86b7d" (UID: "422cdcf0-bced-4e1e-89d0-b475f6e86b7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:26.233868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.233852    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.235563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.235537    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q" pod "422cdcf0-bced-4e1e-89d0-b475f6e86b7d" (UID: "422cdcf0-bced-4e1e-89d0-b475f6e86b7d"). InnerVolumeSpecName "kube-api-access-k6l7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:26.335267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.335217    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6l7q\" (UniqueName: \"kubernetes.io/projected/422cdcf0-bced-4e1e-89d0-b475f6e86b7d-kube-api-access-k6l7q\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.657304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.657273    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"
Oct 30 22:15:26.657841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.657272    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw" event={"ID":"de188dba-7ec0-4328-b09c-2f3b9dab348d","Type":"ContainerDied","Data":"6b8b61af2e5bcbfc42a1e880258c7b96782a1bc404bd062bf948c34644a014d0"}
Oct 30 22:15:26.657841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.657434    5028 scope.go:122] "RemoveContainer" containerID="3910a69b8c02a07d0cbcad0f9fe5ef7c099356464732b30d95b84777de87f358"
Oct 30 22:15:26.658898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.658870    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7" event={"ID":"422cdcf0-bced-4e1e-89d0-b475f6e86b7d","Type":"ContainerDied","Data":"7602335d6ff99b5422ec77e1f33122306da0e61e60a31bf261e396e77f755de2"}
Oct 30 22:15:26.659002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.658909    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"
Oct 30 22:15:26.668525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.668507    5028 scope.go:122] "RemoveContainer" containerID="93f3d2f8676014f1c32eb9aa4d4c262ec64a9962352045aedbd0698ca342bb31"
Oct 30 22:15:26.770704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.770657    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"]
Oct 30 22:15:26.821718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.821667    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"]
Oct 30 22:15:26.827996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.827955    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889975-492gw"]
Oct 30 22:15:26.932935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.932902    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"]
Oct 30 22:15:26.933375 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.933356    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:26.935698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:26.935664    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889975-5w7k7"]
Oct 30 22:15:27.044237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.044200    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g\" (UniqueName: \"kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g\") pod \"c556a798-9568-429a-b952-c380629cc322\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") "
Oct 30 22:15:27.044422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.044312    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca\") pod \"c556a798-9568-429a-b952-c380629cc322\" (UID: \"c556a798-9568-429a-b952-c380629cc322\") "
Oct 30 22:15:27.044703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.044652    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca" pod "c556a798-9568-429a-b952-c380629cc322" (UID: "c556a798-9568-429a-b952-c380629cc322"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:27.047019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.046989    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g" pod "c556a798-9568-429a-b952-c380629cc322" (UID: "c556a798-9568-429a-b952-c380629cc322"). InnerVolumeSpecName "kube-api-access-dvc2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:27.078113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.078071    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.145859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.145825    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c556a798-9568-429a-b952-c380629cc322-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:27.145859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.145852    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvc2g\" (UniqueName: \"kubernetes.io/projected/c556a798-9568-429a-b952-c380629cc322-kube-api-access-dvc2g\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:27.290631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.290594    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" path="/var/lib/kubelet/pods/422cdcf0-bced-4e1e-89d0-b475f6e86b7d/volumes"
Oct 30 22:15:27.291604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.291582    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" path="/var/lib/kubelet/pods/de188dba-7ec0-4328-b09c-2f3b9dab348d/volumes"
Oct 30 22:15:27.666395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.666318    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"
Oct 30 22:15:27.666395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.666366    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg" event={"ID":"c556a798-9568-429a-b952-c380629cc322","Type":"ContainerDied","Data":"cc547d54e0ebdc941d51012799a8300129462e309f24033aefa3f515cd7cce8b"}
Oct 30 22:15:27.666907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.666433    5028 scope.go:122] "RemoveContainer" containerID="5389e85001657880407ac4ed078ee900c2abba27e84f33480f9f4077bd4b677c"
Oct 30 22:15:27.827602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.827563    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"]
Oct 30 22:15:27.835672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:27.835635    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889975-6splg"]
Oct 30 22:15:28.437414 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:28.437366    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:15:29.294138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.294094    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c556a798-9568-429a-b952-c380629cc322" path="/var/lib/kubelet/pods/c556a798-9568-429a-b952-c380629cc322/volumes"
Oct 30 22:15:29.790708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.790650    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"]
Oct 30 22:15:29.791176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791162    5028 cpu_manager.go:414] "containerMap: removing container" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.791238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791192    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.791238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791213    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.791238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791218    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.791238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791226    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.791238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791233    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791246    5028 cpu_manager.go:414] "containerMap: removing container" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791251    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791256    5028 cpu_manager.go:414] "containerMap: removing container" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791261    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791271    5028 cpu_manager.go:414] "containerMap: removing container" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791275    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791283    5028 cpu_manager.go:414] "containerMap: removing container" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791287    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791300    5028 cpu_manager.go:414] "containerMap: removing container" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791304    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791313    5028 cpu_manager.go:414] "containerMap: removing container" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791319    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791332    5028 cpu_manager.go:414] "containerMap: removing container" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791340    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791350    5028 cpu_manager.go:414] "containerMap: removing container" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791354    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791362    5028 cpu_manager.go:414] "containerMap: removing container" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791366    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791373    5028 cpu_manager.go:414] "containerMap: removing container" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791377    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791388    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.791462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791392    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791485    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791497    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791506    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791512    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791517    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791523    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791528    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c556a798-9568-429a-b952-c380629cc322" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791537    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791542    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791547    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791553    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="422cdcf0-bced-4e1e-89d0-b475f6e86b7d" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791559    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d33543-6a75-4d8c-ac47-a5674327f173" containerName="cleanup"
Oct 30 22:15:29.792126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.791569    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:29.796462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.796443    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.850909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.850873    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"]
Oct 30 22:15:29.872113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.872075    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.872310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.872141    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.872310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.872288    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.872420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.872397    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.872486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.872474    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.872532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.872497    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpk2\" (UniqueName: \"kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.894152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.894116    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"premium-w6-be17e7-kserve-self-signed-certs\""
Oct 30 22:15:29.894516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.894497    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"kube-root-ca.crt\""
Oct 30 22:15:29.895341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.895320    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"openshift-service-ca.crt\""
Oct 30 22:15:29.895582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.895567    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"default-dockercfg-vw7ld\""
Oct 30 22:15:29.899058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.897535    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:29.973785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.973667    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.973995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.973834    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.973995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.973858    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpk2\" (UniqueName: \"kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.973995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.973881    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.973995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.973909    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.974227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.974000    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.974227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.974116    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.974472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.974447    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.974472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.974463    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.976378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.976346    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:29.976538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:29.976517    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:30.033211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.033171    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpk2\" (UniqueName: \"kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2\") pod \"premium-w6-be17e7-kserve-77f4b7f5f-4xc4m\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") " pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:30.108936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.108842    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:30.524581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.524552    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"]
Oct 30 22:15:30.525644 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:30.525618    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb5ee88_1511_42b1_8f5f_2f5eed4ee789.slice/crio-605ada37cf8426d4399357a795c4c5f6e338cfea93f35ba1fb742f6c928d6775 WatchSource:0}: Error finding container 605ada37cf8426d4399357a795c4c5f6e338cfea93f35ba1fb742f6c928d6775: Status 404 returned error can't find the container with id 605ada37cf8426d4399357a795c4c5f6e338cfea93f35ba1fb742f6c928d6775
Oct 30 22:15:30.576537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.576506    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"]
Oct 30 22:15:30.577141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.577125    5028 cpu_manager.go:414] "containerMap: removing container" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:30.577197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.577143    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:30.577262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.577252    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a1c655-faa9-4885-b528-9f33bf75f7a6" containerName="cleanup"
Oct 30 22:15:30.577303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.577265    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="de188dba-7ec0-4328-b09c-2f3b9dab348d" containerName="cleanup"
Oct 30 22:15:30.580445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.580428    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.633182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.633119    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:15:30.633182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.633160    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:30.637174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.637143    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"]
Oct 30 22:15:30.679075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.679035    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:30.679246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.679130    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:30.685329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.685300    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.685453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.685433    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.685586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.685525    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzzh\" (UniqueName: \"kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.692823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.692795    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" event={"ID":"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789","Type":"ContainerStarted","Data":"e164b0aea4856c63ea6774a61f63024c1562d0aa4ad114fea7714d26bf7f87c6"}
Oct 30 22:15:30.692968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.692837    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" event={"ID":"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789","Type":"ContainerStarted","Data":"605ada37cf8426d4399357a795c4c5f6e338cfea93f35ba1fb742f6c928d6775"}
Oct 30 22:15:30.787111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.787017    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzzh\" (UniqueName: \"kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.787111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.787104    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.787364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.787167    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.787899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.787877    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.789564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.789545    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.850628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.850599    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzzh\" (UniqueName: \"kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh\") pod \"maas-api-revoke-sub-d7cee2dd-d772p\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:30.857343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.857293    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podStartSLOduration=1.857277817 podStartE2EDuration="1.857277817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:30.854304163 +0000 UTC m=+861.703817136" watchObservedRunningTime="2026-10-30 22:15:30.857277817 +0000 UTC m=+861.706790827"
Oct 30 22:15:30.897452 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:30.897415    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:31.109306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:31.109218    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:31.273204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:31.273172    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"]
Oct 30 22:15:31.274200 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:31.274170    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86e4bc2_746c_4f3b_aa01_653c1cbbe71d.slice/crio-9116af6d3569970dd0c2d4faa12ae026302c3013f660004def2e66fe102bc23b WatchSource:0}: Error finding container 9116af6d3569970dd0c2d4faa12ae026302c3013f660004def2e66fe102bc23b: Status 404 returned error can't find the container with id 9116af6d3569970dd0c2d4faa12ae026302c3013f660004def2e66fe102bc23b
Oct 30 22:15:31.701364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:31.701265    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p" event={"ID":"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d","Type":"ContainerStarted","Data":"bee05b1c1e3c6379d1ad3a13b7a3e3322d1af7a63efb7316ffce96d37f2dfab6"}
Oct 30 22:15:31.701364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:31.701329    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p" event={"ID":"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d","Type":"ContainerStarted","Data":"9116af6d3569970dd0c2d4faa12ae026302c3013f660004def2e66fe102bc23b"}
Oct 30 22:15:31.813725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:31.813650    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p" podStartSLOduration=1.813635916 podStartE2EDuration="1.813635916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:31.812164622 +0000 UTC m=+862.661677577" watchObservedRunningTime="2026-10-30 22:15:31.813635916 +0000 UTC m=+862.663148888"
Oct 30 22:15:32.091456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:32.091407    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:32.709565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:32.709521    5028 generic.go:356] "Generic (PLEG): container finished" podID="a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" containerID="bee05b1c1e3c6379d1ad3a13b7a3e3322d1af7a63efb7316ffce96d37f2dfab6" exitCode=0
Oct 30 22:15:32.709970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:32.709605    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p" event={"ID":"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d","Type":"ContainerDied","Data":"bee05b1c1e3c6379d1ad3a13b7a3e3322d1af7a63efb7316ffce96d37f2dfab6"}
Oct 30 22:15:33.876210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:33.876180    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:34.020020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.019934    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca\") pod \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") "
Oct 30 22:15:34.020192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.020076    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca\") pod \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") "
Oct 30 22:15:34.020265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.020202    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh\" (UniqueName: \"kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh\") pod \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\" (UID: \"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d\") "
Oct 30 22:15:34.020516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.020484    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca" pod "a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" (UID: "a86e4bc2-746c-4f3b-aa01-653c1cbbe71d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:34.022330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.022302    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh" pod "a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" (UID: "a86e4bc2-746c-4f3b-aa01-653c1cbbe71d"). InnerVolumeSpecName "kube-api-access-lkzzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:34.022459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.022438    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca" pod "a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" (UID: "a86e4bc2-746c-4f3b-aa01-653c1cbbe71d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:34.121611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.121570    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.121611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.121603    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkzzh\" (UniqueName: \"kubernetes.io/projected/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-kube-api-access-lkzzh\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.121611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.121617    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:34.727283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.727249    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"
Oct 30 22:15:34.727512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.727285    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p" event={"ID":"a86e4bc2-746c-4f3b-aa01-653c1cbbe71d","Type":"ContainerDied","Data":"9116af6d3569970dd0c2d4faa12ae026302c3013f660004def2e66fe102bc23b"}
Oct 30 22:15:34.727512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:34.727332    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9116af6d3569970dd0c2d4faa12ae026302c3013f660004def2e66fe102bc23b"
Oct 30 22:15:37.099776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:37.099739    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:40.109393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:40.109354    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:40.122042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:40.122015    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:40.790501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:40.790469    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:15:41.943920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:41.943883    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:45.610796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:45.610763    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:45.760182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:45.760143    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:46.883529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:46.883475    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:51.780242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:51.780193    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:53.291260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.291224    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"]
Oct 30 22:15:53.291739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.291724    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" containerName="revoke-keys"
Oct 30 22:15:53.291790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.291740    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" containerName="revoke-keys"
Oct 30 22:15:53.291840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.291830    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" containerName="revoke-keys"
Oct 30 22:15:53.299323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.299298    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.339758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.337416    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.339758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.337482    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.339758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.337555    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.339758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.337625    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.339758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.337680    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.339758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.337846    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6f2\" (UniqueName: \"kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.355680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.355642    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"]
Oct 30 22:15:53.380909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.380870    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:53.395126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.395091    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"openshift-service-ca.crt\""
Oct 30 22:15:53.395126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.395107    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"default-dockercfg-2zk59\""
Oct 30 22:15:53.395340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.395176    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"premium-w4-b13755-kserve-self-signed-certs\""
Oct 30 22:15:53.397657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.397637    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"kube-root-ca.crt\""
Oct 30 22:15:53.439044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439005    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439094    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6f2\" (UniqueName: \"kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439143    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439212    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439265    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439322    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439746    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439789    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.439919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.439838    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.441921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.441894    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.442227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.442208    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.498638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.498601    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6f2\" (UniqueName: \"kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2\") pod \"premium-w4-b13755-kserve-6bfdbf589-6kz2n\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") " pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:53.615744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:53.615383    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:54.018572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:54.018532    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"]
Oct 30 22:15:54.018839 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:15:54.018817    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65ebfd05_06fb_476e_9d13_30a28e9bfc4a.slice/crio-375e81ea9af305af4e7abbc952df67838909769237cb1074508aeb4d6ebae12f WatchSource:0}: Error finding container 375e81ea9af305af4e7abbc952df67838909769237cb1074508aeb4d6ebae12f: Status 404 returned error can't find the container with id 375e81ea9af305af4e7abbc952df67838909769237cb1074508aeb4d6ebae12f
Oct 30 22:15:54.872355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:54.872316    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" event={"ID":"65ebfd05-06fb-476e-9d13-30a28e9bfc4a","Type":"ContainerStarted","Data":"c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96"}
Oct 30 22:15:54.872886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:54.872370    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" event={"ID":"65ebfd05-06fb-476e-9d13-30a28e9bfc4a","Type":"ContainerStarted","Data":"375e81ea9af305af4e7abbc952df67838909769237cb1074508aeb4d6ebae12f"}
Oct 30 22:15:54.986384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:54.986330    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podStartSLOduration=1.986315102 podStartE2EDuration="1.986315102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:54.984241608 +0000 UTC m=+885.833754584" watchObservedRunningTime="2026-10-30 22:15:54.986315102 +0000 UTC m=+885.835828075"
Oct 30 22:15:55.616413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:55.616368    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:15:56.695039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:15:56.694952    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:58.435485 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:15:58.435443    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:16:00.950662 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:00.950624    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"]
Oct 30 22:16:00.955041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:00.955015    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.031012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.030964    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj99z\" (UniqueName: \"kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.031232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.031082    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.031300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.031269    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.053610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.053569    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:01.053610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.053569    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:16:01.053610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.053574    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:01.053884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.053724    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:01.056694 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.056662    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"]
Oct 30 22:16:01.132264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.132227    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.132447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.132281    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj99z\" (UniqueName: \"kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.132447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.132332    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.132982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.132950    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.134763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.134740    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.194263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.194228    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj99z\" (UniqueName: \"kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z\") pod \"maas-api-revoke-sub-e6832394-2b48g\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.269395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.269353    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:01.593710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.593606    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:01.650674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.650641    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"]
Oct 30 22:16:01.650915 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:16:01.650889    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e44a52_3b3e_448e_9dc8_a55417a57a56.slice/crio-2ded0d08e0ebb4d8b161e4736b8d9dc561601fa58f99ad61d7271ebedb6b1016 WatchSource:0}: Error finding container 2ded0d08e0ebb4d8b161e4736b8d9dc561601fa58f99ad61d7271ebedb6b1016: Status 404 returned error can't find the container with id 2ded0d08e0ebb4d8b161e4736b8d9dc561601fa58f99ad61d7271ebedb6b1016
Oct 30 22:16:01.918748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:01.918717    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g" event={"ID":"f4e44a52-3b3e-448e-9dc8-a55417a57a56","Type":"ContainerStarted","Data":"2ded0d08e0ebb4d8b161e4736b8d9dc561601fa58f99ad61d7271ebedb6b1016"}
Oct 30 22:16:02.351317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.351265    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"]
Oct 30 22:16:02.357940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.357913    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.408242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.408201    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f80a0e-openshift-default-dockercfg-h5r64\""
Oct 30 22:16:02.456983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.456944    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.457172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457027    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.457172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457137    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.458813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457181    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.458813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457356    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.458813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457386    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zvc\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.458813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457429    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.458813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.457568    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.458813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.458778    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.459870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.459848    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"]
Oct 30 22:16:02.560248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560205    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560262    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560302    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560347    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560371    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560393    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560725    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560768    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560802    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zvc\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560848    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.560924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560900    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.561032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.560994    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.561127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.561101    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.561616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.561597    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.563122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.563096    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.563304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.563283    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.619831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.619752    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.619831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.619754    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zvc\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.673263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.673224    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:02.929327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.929231    5028 generic.go:356] "Generic (PLEG): container finished" podID="f4e44a52-3b3e-448e-9dc8-a55417a57a56" containerID="5c1e4780a805dfcd8d065d2924802fb801574c3351912312882fcbf214b0a2c3" exitCode=0
Oct 30 22:16:02.929488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:02.929322    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g" event={"ID":"f4e44a52-3b3e-448e-9dc8-a55417a57a56","Type":"ContainerDied","Data":"5c1e4780a805dfcd8d065d2924802fb801574c3351912312882fcbf214b0a2c3"}
Oct 30 22:16:03.171610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.171577    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"]
Oct 30 22:16:03.202748 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:16:03.202716    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fce23ee_ef26_43d8_90fb_be6d1252a2e2.slice/crio-ad30514901f10c0e3855dfd05f2028726bae3af37ced485ea1e801e1c4179d49 WatchSource:0}: Error finding container ad30514901f10c0e3855dfd05f2028726bae3af37ced485ea1e801e1c4179d49: Status 404 returned error can't find the container with id ad30514901f10c0e3855dfd05f2028726bae3af37ced485ea1e801e1c4179d49
Oct 30 22:16:03.205214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.205180    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:03.205311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.205270    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:03.205381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.205313    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:03.616499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.616461    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:16:03.635997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.635968    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:16:03.964775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.964740    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" event={"ID":"6fce23ee-ef26-43d8-90fb-be6d1252a2e2","Type":"ContainerStarted","Data":"a720fb40ecabf6a34839d6e2c884966aac4b4cf7647e2a90d204c448365bf601"}
Oct 30 22:16:03.964992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.964802    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" event={"ID":"6fce23ee-ef26-43d8-90fb-be6d1252a2e2","Type":"ContainerStarted","Data":"ad30514901f10c0e3855dfd05f2028726bae3af37ced485ea1e801e1c4179d49"}
Oct 30 22:16:03.979717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:03.979674    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:16:04.086102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.086043    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podStartSLOduration=2.086025111 podStartE2EDuration="2.086025111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:04.084720305 +0000 UTC m=+894.934233279" watchObservedRunningTime="2026-10-30 22:16:04.086025111 +0000 UTC m=+894.935538087"
Oct 30 22:16:04.169966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.169935    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:04.184802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.184778    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:04.284839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.284755    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca\") pod \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") "
Oct 30 22:16:04.284839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.284804    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z\" (UniqueName: \"kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z\") pod \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") "
Oct 30 22:16:04.285082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.284882    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca\") pod \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\" (UID: \"f4e44a52-3b3e-448e-9dc8-a55417a57a56\") "
Oct 30 22:16:04.285718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.285675    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca" pod "f4e44a52-3b3e-448e-9dc8-a55417a57a56" (UID: "f4e44a52-3b3e-448e-9dc8-a55417a57a56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:04.287147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.287114    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca" pod "f4e44a52-3b3e-448e-9dc8-a55417a57a56" (UID: "f4e44a52-3b3e-448e-9dc8-a55417a57a56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:04.287793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.287763    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z" pod "f4e44a52-3b3e-448e-9dc8-a55417a57a56" (UID: "f4e44a52-3b3e-448e-9dc8-a55417a57a56"). InnerVolumeSpecName "kube-api-access-dj99z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:04.386732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.386227    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.386732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.386262    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e44a52-3b3e-448e-9dc8-a55417a57a56-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.386732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.386277    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj99z\" (UniqueName: \"kubernetes.io/projected/f4e44a52-3b3e-448e-9dc8-a55417a57a56-kube-api-access-dj99z\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:04.673532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.673433    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:04.675262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.675237    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.675381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.675295    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:16:04.971514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.971471    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g" event={"ID":"f4e44a52-3b3e-448e-9dc8-a55417a57a56","Type":"ContainerDied","Data":"2ded0d08e0ebb4d8b161e4736b8d9dc561601fa58f99ad61d7271ebedb6b1016"}
Oct 30 22:16:04.971733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.971533    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ded0d08e0ebb4d8b161e4736b8d9dc561601fa58f99ad61d7271ebedb6b1016"
Oct 30 22:16:04.971733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:04.971500    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"
Oct 30 22:16:05.673619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:05.673582    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.674052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:05.673638    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:16:06.503183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:06.503149    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:06.677139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:06.677103    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.677747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:06.677714    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:16:07.674231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:07.674187    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.674558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:07.674522    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:16:08.674303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:08.674262    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.674820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:08.674321    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:16:09.440564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:09.440537    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:16:09.441870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:09.441841    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:16:09.673609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:09.673573    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.673827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:09.673632    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 22:16:10.678534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:10.678500    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:10.679029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:10.678908    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:10.679573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:10.679550    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:16:11.477211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:11.477173    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:15.845896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:15.845852    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:16.382845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.382810    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:16.566864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.566821    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"]
Oct 30 22:16:16.567393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.567379    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f4e44a52-3b3e-448e-9dc8-a55417a57a56" containerName="revoke-keys"
Oct 30 22:16:16.567436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.567395    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e44a52-3b3e-448e-9dc8-a55417a57a56" containerName="revoke-keys"
Oct 30 22:16:16.567494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.567485    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e44a52-3b3e-448e-9dc8-a55417a57a56" containerName="revoke-keys"
Oct 30 22:16:16.572850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.572827    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.626144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.626114    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"]
Oct 30 22:16:16.668497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.668402    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:16.668497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.668454    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:16.671728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.671704    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:16.671728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.671727    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:16:16.672121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.672096    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.672312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.672296    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.672366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.672326    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dcjx\" (UniqueName: \"kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.773229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.773194    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.773229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.773237    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dcjx\" (UniqueName: \"kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.773500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.773311    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.774150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.774125    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.776074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.776047    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.833592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.833560    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dcjx\" (UniqueName: \"kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx\") pod \"maas-api-revoke-sub-90d724c6-2t8b4\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:16.886317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:16.886284    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:17.266750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:17.262626    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"]
Oct 30 22:16:17.268668 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:16:17.268632    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21137bf_ef4a_4233_84b4_29555de5090c.slice/crio-68fdbc783641a033412e9dac8e8cd9a64486f55e2669cedb83f3e556584f71b8 WatchSource:0}: Error finding container 68fdbc783641a033412e9dac8e8cd9a64486f55e2669cedb83f3e556584f71b8: Status 404 returned error can't find the container with id 68fdbc783641a033412e9dac8e8cd9a64486f55e2669cedb83f3e556584f71b8
Oct 30 22:16:18.072480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:18.072438    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4" event={"ID":"a21137bf-ef4a-4233-84b4-29555de5090c","Type":"ContainerStarted","Data":"910e865216405b3416daf487ddcc4b1dbf0bf2ce2836572d98b56d5c6ca6bfbc"}
Oct 30 22:16:18.072905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:18.072508    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4" event={"ID":"a21137bf-ef4a-4233-84b4-29555de5090c","Type":"ContainerStarted","Data":"68fdbc783641a033412e9dac8e8cd9a64486f55e2669cedb83f3e556584f71b8"}
Oct 30 22:16:18.189604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:18.189550    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4" podStartSLOduration=2.189536722 podStartE2EDuration="2.189536722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:18.186930592 +0000 UTC m=+909.036443544" watchObservedRunningTime="2026-10-30 22:16:18.189536722 +0000 UTC m=+909.039049694"
Oct 30 22:16:19.081608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:19.081569    5028 generic.go:356] "Generic (PLEG): container finished" podID="a21137bf-ef4a-4233-84b4-29555de5090c" containerID="910e865216405b3416daf487ddcc4b1dbf0bf2ce2836572d98b56d5c6ca6bfbc" exitCode=0
Oct 30 22:16:19.082111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:19.081655    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4" event={"ID":"a21137bf-ef4a-4233-84b4-29555de5090c","Type":"ContainerDied","Data":"910e865216405b3416daf487ddcc4b1dbf0bf2ce2836572d98b56d5c6ca6bfbc"}
Oct 30 22:16:20.251064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.251033    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:20.311624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.311583    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca\") pod \"a21137bf-ef4a-4233-84b4-29555de5090c\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") "
Oct 30 22:16:20.311624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.311626    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca\") pod \"a21137bf-ef4a-4233-84b4-29555de5090c\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") "
Oct 30 22:16:20.311916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.311733    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx\" (UniqueName: \"kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx\") pod \"a21137bf-ef4a-4233-84b4-29555de5090c\" (UID: \"a21137bf-ef4a-4233-84b4-29555de5090c\") "
Oct 30 22:16:20.312035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.312007    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca" pod "a21137bf-ef4a-4233-84b4-29555de5090c" (UID: "a21137bf-ef4a-4233-84b4-29555de5090c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:20.313881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.313843    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca" pod "a21137bf-ef4a-4233-84b4-29555de5090c" (UID: "a21137bf-ef4a-4233-84b4-29555de5090c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:20.314020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.313975    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx" pod "a21137bf-ef4a-4233-84b4-29555de5090c" (UID: "a21137bf-ef4a-4233-84b4-29555de5090c"). InnerVolumeSpecName "kube-api-access-8dcjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:20.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.413625    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dcjx\" (UniqueName: \"kubernetes.io/projected/a21137bf-ef4a-4233-84b4-29555de5090c-kube-api-access-8dcjx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.413654    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:20.413666    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21137bf-ef4a-4233-84b4-29555de5090c-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:21.096222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:21.096186    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4" event={"ID":"a21137bf-ef4a-4233-84b4-29555de5090c","Type":"ContainerDied","Data":"68fdbc783641a033412e9dac8e8cd9a64486f55e2669cedb83f3e556584f71b8"}
Oct 30 22:16:21.096382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:21.096237    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68fdbc783641a033412e9dac8e8cd9a64486f55e2669cedb83f3e556584f71b8"
Oct 30 22:16:21.096382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:21.096199    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"
Oct 30 22:16:21.272055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:21.272013    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:26.237977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:26.237932    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.435805 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:16:28.435766    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:16:31.107530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:31.107491    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:32.443373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.443330    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"]
Oct 30 22:16:32.444143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.444112    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a21137bf-ef4a-4233-84b4-29555de5090c" containerName="revoke-keys"
Oct 30 22:16:32.444248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.444144    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21137bf-ef4a-4233-84b4-29555de5090c" containerName="revoke-keys"
Oct 30 22:16:32.444329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.444316    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21137bf-ef4a-4233-84b4-29555de5090c" containerName="revoke-keys"
Oct 30 22:16:32.447826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.447805    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.497161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.497112    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.497337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.497188    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.497337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.497301    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6mds\" (UniqueName: \"kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.498193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.498167    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:32.545663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.545624    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:16:32.545909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.545677    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:32.545909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.545701    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:32.549151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.549124    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"]
Oct 30 22:16:32.598571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.598532    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.598799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.598710    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.598799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.598776    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6mds\" (UniqueName: \"kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.599381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.599352    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.600938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.600919    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.658101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.658058    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6mds\" (UniqueName: \"kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds\") pod \"maas-api-revoke-sub-f140febb-bmmxv\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:32.763730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:32.763666    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:33.141493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:33.141460    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"]
Oct 30 22:16:33.142070 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:16:33.142036    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe24d802_13f9_46de_ae2e_07becd46d6bb.slice/crio-92033e9cab5d63a151e823296139f5fd4bba5147bcfe0ca4521aadb3cfc73c65 WatchSource:0}: Error finding container 92033e9cab5d63a151e823296139f5fd4bba5147bcfe0ca4521aadb3cfc73c65: Status 404 returned error can't find the container with id 92033e9cab5d63a151e823296139f5fd4bba5147bcfe0ca4521aadb3cfc73c65
Oct 30 22:16:33.192203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:33.192163    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv" event={"ID":"be24d802-13f9-46de-ae2e-07becd46d6bb","Type":"ContainerStarted","Data":"92033e9cab5d63a151e823296139f5fd4bba5147bcfe0ca4521aadb3cfc73c65"}
Oct 30 22:16:34.200847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:34.200804    5028 generic.go:356] "Generic (PLEG): container finished" podID="be24d802-13f9-46de-ae2e-07becd46d6bb" containerID="7ac774904203990bda895b17cdbf38734aa41a08929ec1932887f675faacda78" exitCode=0
Oct 30 22:16:34.201325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:34.200869    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv" event={"ID":"be24d802-13f9-46de-ae2e-07becd46d6bb","Type":"ContainerDied","Data":"7ac774904203990bda895b17cdbf38734aa41a08929ec1932887f675faacda78"}
Oct 30 22:16:35.374035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.374004    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:35.429464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.429427    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca\") pod \"be24d802-13f9-46de-ae2e-07becd46d6bb\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") "
Oct 30 22:16:35.429623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.429517    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca\") pod \"be24d802-13f9-46de-ae2e-07becd46d6bb\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") "
Oct 30 22:16:35.429623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.429596    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds\" (UniqueName: \"kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds\") pod \"be24d802-13f9-46de-ae2e-07becd46d6bb\" (UID: \"be24d802-13f9-46de-ae2e-07becd46d6bb\") "
Oct 30 22:16:35.429885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.429861    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca" pod "be24d802-13f9-46de-ae2e-07becd46d6bb" (UID: "be24d802-13f9-46de-ae2e-07becd46d6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:35.430050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.430032    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:35.431721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.431679    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca" pod "be24d802-13f9-46de-ae2e-07becd46d6bb" (UID: "be24d802-13f9-46de-ae2e-07becd46d6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:35.431826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.431727    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds" pod "be24d802-13f9-46de-ae2e-07becd46d6bb" (UID: "be24d802-13f9-46de-ae2e-07becd46d6bb"). InnerVolumeSpecName "kube-api-access-j6mds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:35.530678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.530645    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be24d802-13f9-46de-ae2e-07becd46d6bb-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:35.530678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:35.530674    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6mds\" (UniqueName: \"kubernetes.io/projected/be24d802-13f9-46de-ae2e-07becd46d6bb-kube-api-access-j6mds\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:36.081352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:36.081318    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.214732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:36.214680    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv" event={"ID":"be24d802-13f9-46de-ae2e-07becd46d6bb","Type":"ContainerDied","Data":"92033e9cab5d63a151e823296139f5fd4bba5147bcfe0ca4521aadb3cfc73c65"}
Oct 30 22:16:36.214913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:36.214752    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92033e9cab5d63a151e823296139f5fd4bba5147bcfe0ca4521aadb3cfc73c65"
Oct 30 22:16:36.214913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:36.214708    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"
Oct 30 22:16:40.895279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:40.895246    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:45.896540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:45.896498    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:46.620818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.620778    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"]
Oct 30 22:16:46.621477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.621453    5028 cpu_manager.go:414] "containerMap: removing container" podUID="be24d802-13f9-46de-ae2e-07becd46d6bb" containerName="revoke-keys"
Oct 30 22:16:46.621477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.621474    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be24d802-13f9-46de-ae2e-07becd46d6bb" containerName="revoke-keys"
Oct 30 22:16:46.621736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.621721    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="be24d802-13f9-46de-ae2e-07becd46d6bb" containerName="revoke-keys"
Oct 30 22:16:46.625464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.625443    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.727041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.727006    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"]
Oct 30 22:16:46.758383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758349    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758396    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758545    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758609    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758637    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758657    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758677    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zg2\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758851    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.758964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.758935    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.859890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.859849    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zg2\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.859946    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.859986    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860045    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860088    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860156    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860189    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860233    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860328    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860571    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860629    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860807    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.860863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.860829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.861129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.861103    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.862582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.862555    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.862752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.862734    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.919195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.919117    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.919540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.919259    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zg2\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:46.940075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:46.940040    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:47.362012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:47.361978    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"]
Oct 30 22:16:47.362988 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:16:47.362952    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106fddfb_ab86_488e_9d28_31a4b45a7bdc.slice/crio-75b9b0a66e056867ebb7ad8ee93192755048c6a95d461a1767fd5cd8e1c18f8d WatchSource:0}: Error finding container 75b9b0a66e056867ebb7ad8ee93192755048c6a95d461a1767fd5cd8e1c18f8d: Status 404 returned error can't find the container with id 75b9b0a66e056867ebb7ad8ee93192755048c6a95d461a1767fd5cd8e1c18f8d
Oct 30 22:16:47.365460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:47.365427    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:47.365560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:47.365497    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:47.365560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:47.365525    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:48.301899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:48.301860    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" event={"ID":"106fddfb-ab86-488e-9d28-31a4b45a7bdc","Type":"ContainerStarted","Data":"71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a"}
Oct 30 22:16:48.302327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:48.301925    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" event={"ID":"106fddfb-ab86-488e-9d28-31a4b45a7bdc","Type":"ContainerStarted","Data":"75b9b0a66e056867ebb7ad8ee93192755048c6a95d461a1767fd5cd8e1c18f8d"}
Oct 30 22:16:48.520076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:48.516648    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podStartSLOduration=2.516632767 podStartE2EDuration="2.516632767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:48.468254491 +0000 UTC m=+939.317767466" watchObservedRunningTime="2026-10-30 22:16:48.516632767 +0000 UTC m=+939.366146091"
Oct 30 22:16:48.943094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:48.943047    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:48.944860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:48.944828    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:16:48.945019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:48.944878    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 22:16:49.947713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:49.944458    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:16:49.947713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:49.944544    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 22:16:50.768530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:50.768452    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:50.941715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:50.941103    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:16:50.941715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:50.941171    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 22:16:51.942254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:51.942212    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:16:51.942746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:51.942283    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 22:16:52.867537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.867500    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"]
Oct 30 22:16:52.872134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.872106    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:52.930875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.930842    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:52.932006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.931329    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:16:52.941656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.941222    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:16:52.941656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.941291    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 22:16:52.947653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.947621    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"]
Oct 30 22:16:52.962811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.962315    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:52.962811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.962461    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8df29\" (UniqueName: \"kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:52.962811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.962745    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:52.970442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.970405    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:52.970616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:52.970457    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:53.064251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.064217    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.064565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.064544    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8df29\" (UniqueName: \"kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.064865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.064851    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.065629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.065610    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.067342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.067323    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.144760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.143514    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8df29\" (UniqueName: \"kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29\") pod \"maas-api-revoke-sub-fdda3e8b-rbwh4\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.186290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.186259    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:53.587341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:53.587311    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"]
Oct 30 22:16:53.588960 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:16:53.588933    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2978736e_5cdb_4b19_ba7a_ad66b2cb6088.slice/crio-d68e999b4641c25faa118f5ea2828dbf8f7454ad750b98a2c7e810379a4f2fb0 WatchSource:0}: Error finding container d68e999b4641c25faa118f5ea2828dbf8f7454ad750b98a2c7e810379a4f2fb0: Status 404 returned error can't find the container with id d68e999b4641c25faa118f5ea2828dbf8f7454ad750b98a2c7e810379a4f2fb0
Oct 30 22:16:54.096406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:54.095013    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:54.096406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:54.096293    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:54.097122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:54.097101    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:16:54.381676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:54.381585    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4" event={"ID":"2978736e-5cdb-4b19-ba7a-ad66b2cb6088","Type":"ContainerStarted","Data":"452ad910545e650d07fdb9c1c95be48c196f11fb20ee0bf5ddc9a0a0fdc6fec7"}
Oct 30 22:16:54.381676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:54.381648    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4" event={"ID":"2978736e-5cdb-4b19-ba7a-ad66b2cb6088","Type":"ContainerStarted","Data":"d68e999b4641c25faa118f5ea2828dbf8f7454ad750b98a2c7e810379a4f2fb0"}
Oct 30 22:16:54.498503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:54.498438    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4" podStartSLOduration=2.498421434 podStartE2EDuration="2.498421434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:54.494327217 +0000 UTC m=+945.343840291" watchObservedRunningTime="2026-10-30 22:16:54.498421434 +0000 UTC m=+945.347934407"
Oct 30 22:16:55.390238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:55.390202    5028 generic.go:356] "Generic (PLEG): container finished" podID="2978736e-5cdb-4b19-ba7a-ad66b2cb6088" containerID="452ad910545e650d07fdb9c1c95be48c196f11fb20ee0bf5ddc9a0a0fdc6fec7" exitCode=0
Oct 30 22:16:55.390674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:55.390285    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4" event={"ID":"2978736e-5cdb-4b19-ba7a-ad66b2cb6088","Type":"ContainerDied","Data":"452ad910545e650d07fdb9c1c95be48c196f11fb20ee0bf5ddc9a0a0fdc6fec7"}
Oct 30 22:16:55.593252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:55.593219    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:56.556934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.556897    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:56.606208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.606167    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca\") pod \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") "
Oct 30 22:16:56.606432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.606245    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29\" (UniqueName: \"kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29\") pod \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") "
Oct 30 22:16:56.606432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.606306    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca\") pod \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\" (UID: \"2978736e-5cdb-4b19-ba7a-ad66b2cb6088\") "
Oct 30 22:16:56.606708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.606654    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca" pod "2978736e-5cdb-4b19-ba7a-ad66b2cb6088" (UID: "2978736e-5cdb-4b19-ba7a-ad66b2cb6088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:56.608539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.608516    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca" pod "2978736e-5cdb-4b19-ba7a-ad66b2cb6088" (UID: "2978736e-5cdb-4b19-ba7a-ad66b2cb6088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:56.608650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.608628    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29" pod "2978736e-5cdb-4b19-ba7a-ad66b2cb6088" (UID: "2978736e-5cdb-4b19-ba7a-ad66b2cb6088"). InnerVolumeSpecName "kube-api-access-8df29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:56.707283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.707178    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8df29\" (UniqueName: \"kubernetes.io/projected/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-kube-api-access-8df29\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:56.707283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.707216    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:56.707283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:56.707232    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978736e-5cdb-4b19-ba7a-ad66b2cb6088-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:57.404172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:57.404140    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"
Oct 30 22:16:57.404393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:57.404167    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4" event={"ID":"2978736e-5cdb-4b19-ba7a-ad66b2cb6088","Type":"ContainerDied","Data":"d68e999b4641c25faa118f5ea2828dbf8f7454ad750b98a2c7e810379a4f2fb0"}
Oct 30 22:16:57.404393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:16:57.404219    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68e999b4641c25faa118f5ea2828dbf8f7454ad750b98a2c7e810379a4f2fb0"
Oct 30 22:16:58.455308 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:16:58.455270    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:17:00.508282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:00.508241    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:03.666373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.666336    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"]
Oct 30 22:17:03.666954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.666938    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2978736e-5cdb-4b19-ba7a-ad66b2cb6088" containerName="revoke-keys"
Oct 30 22:17:03.666999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.666955    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2978736e-5cdb-4b19-ba7a-ad66b2cb6088" containerName="revoke-keys"
Oct 30 22:17:03.667076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.667067    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2978736e-5cdb-4b19-ba7a-ad66b2cb6088" containerName="revoke-keys"
Oct 30 22:17:03.671652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.671631    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.694363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.694328    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.694528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.694371    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.694528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.694403    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ngz\" (UniqueName: \"kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.773655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.773594    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:17:03.773655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.773609    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:03.773655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.773644    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:17:03.773994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.773733    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:17:03.779378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.779340    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"]
Oct 30 22:17:03.795767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.795728    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.795971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.795775    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.795971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.795799    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ngz\" (UniqueName: \"kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.796498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.796469    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.798194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.798172    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.855904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.855861    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ngz\" (UniqueName: \"kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz\") pod \"maas-api-revoke-sub-40689d5e-gr4vm\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:03.985043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:03.985004    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:04.358482 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:04.358433    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819f57b9_b878_4146_b356_5d341d7625d5.slice/crio-fe96c2a49569fc27abe2e36a76323ab962b061888e046dd10760fd36ad06da75 WatchSource:0}: Error finding container fe96c2a49569fc27abe2e36a76323ab962b061888e046dd10760fd36ad06da75: Status 404 returned error can't find the container with id fe96c2a49569fc27abe2e36a76323ab962b061888e046dd10760fd36ad06da75
Oct 30 22:17:04.359145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:04.359114    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"]
Oct 30 22:17:04.470950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:04.470912    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm" event={"ID":"819f57b9-b878-4146-b356-5d341d7625d5","Type":"ContainerStarted","Data":"fe96c2a49569fc27abe2e36a76323ab962b061888e046dd10760fd36ad06da75"}
Oct 30 22:17:04.517661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:04.517620    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.479039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:05.479004    5028 generic.go:356] "Generic (PLEG): container finished" podID="819f57b9-b878-4146-b356-5d341d7625d5" containerID="af43de7382b5607dc3b650a95ba041639fa21734cd23a3b0f91e1a77cdc5f2b4" exitCode=0
Oct 30 22:17:05.479460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:05.479086    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm" event={"ID":"819f57b9-b878-4146-b356-5d341d7625d5","Type":"ContainerDied","Data":"af43de7382b5607dc3b650a95ba041639fa21734cd23a3b0f91e1a77cdc5f2b4"}
Oct 30 22:17:05.493432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:05.493397    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:06.387663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.387619    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"]
Oct 30 22:17:06.391895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.391869    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.425671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.425633    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.425862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.425776    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.425959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.425942    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjv7\" (UniqueName: \"kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.447640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.447610    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"]
Oct 30 22:17:06.526807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.526766    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.527292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.526899    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjv7\" (UniqueName: \"kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.527292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.526996    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.527446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.527424    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.529502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.529474    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.586915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.586879    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjv7\" (UniqueName: \"kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7\") pod \"maas-api-revoke-sub-73384deb-7p6ss\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.652842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.652774    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:06.705592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.705561    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:06.731175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.731141    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz\" (UniqueName: \"kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz\") pod \"819f57b9-b878-4146-b356-5d341d7625d5\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") "
Oct 30 22:17:06.731303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.731209    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca\") pod \"819f57b9-b878-4146-b356-5d341d7625d5\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") "
Oct 30 22:17:06.731404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.731388    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca\") pod \"819f57b9-b878-4146-b356-5d341d7625d5\" (UID: \"819f57b9-b878-4146-b356-5d341d7625d5\") "
Oct 30 22:17:06.731647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.731616    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca" pod "819f57b9-b878-4146-b356-5d341d7625d5" (UID: "819f57b9-b878-4146-b356-5d341d7625d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:06.731880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.731859    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:06.733594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.733573    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca" pod "819f57b9-b878-4146-b356-5d341d7625d5" (UID: "819f57b9-b878-4146-b356-5d341d7625d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:06.733742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.733727    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz" pod "819f57b9-b878-4146-b356-5d341d7625d5" (UID: "819f57b9-b878-4146-b356-5d341d7625d5"). InnerVolumeSpecName "kube-api-access-j9ngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:06.833334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.833299    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819f57b9-b878-4146-b356-5d341d7625d5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:06.833334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:06.833336    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ngz\" (UniqueName: \"kubernetes.io/projected/819f57b9-b878-4146-b356-5d341d7625d5-kube-api-access-j9ngz\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:07.079141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.079109    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"]
Oct 30 22:17:07.079629 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:07.079601    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551a4fe9_7f02_49a7_86c4_e46f7d30b534.slice/crio-a19e0da14a2adafbe0839dcebea9c7cbaa28d16dd7b1d6c256dc6cc744548667 WatchSource:0}: Error finding container a19e0da14a2adafbe0839dcebea9c7cbaa28d16dd7b1d6c256dc6cc744548667: Status 404 returned error can't find the container with id a19e0da14a2adafbe0839dcebea9c7cbaa28d16dd7b1d6c256dc6cc744548667
Oct 30 22:17:07.496992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.496957    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"
Oct 30 22:17:07.497185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.497000    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm" event={"ID":"819f57b9-b878-4146-b356-5d341d7625d5","Type":"ContainerDied","Data":"fe96c2a49569fc27abe2e36a76323ab962b061888e046dd10760fd36ad06da75"}
Oct 30 22:17:07.497185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.497067    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe96c2a49569fc27abe2e36a76323ab962b061888e046dd10760fd36ad06da75"
Oct 30 22:17:07.500000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.499968    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss" event={"ID":"551a4fe9-7f02-49a7-86c4-e46f7d30b534","Type":"ContainerStarted","Data":"86fd417221393119dac304100b41d305d3bc036584970a4343fb7f85bfd6dda4"}
Oct 30 22:17:07.500135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.500019    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss" event={"ID":"551a4fe9-7f02-49a7-86c4-e46f7d30b534","Type":"ContainerStarted","Data":"a19e0da14a2adafbe0839dcebea9c7cbaa28d16dd7b1d6c256dc6cc744548667"}
Oct 30 22:17:07.614198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:07.614085    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss" podStartSLOduration=1.6140708350000001 podStartE2EDuration="1.614070835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:07.610430154 +0000 UTC m=+958.459943127" watchObservedRunningTime="2026-10-30 22:17:07.614070835 +0000 UTC m=+958.463583808"
Oct 30 22:17:08.525660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:08.525614    5028 generic.go:356] "Generic (PLEG): container finished" podID="551a4fe9-7f02-49a7-86c4-e46f7d30b534" containerID="86fd417221393119dac304100b41d305d3bc036584970a4343fb7f85bfd6dda4" exitCode=0
Oct 30 22:17:08.526011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:08.525982    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss" event={"ID":"551a4fe9-7f02-49a7-86c4-e46f7d30b534","Type":"ContainerDied","Data":"86fd417221393119dac304100b41d305d3bc036584970a4343fb7f85bfd6dda4"}
Oct 30 22:17:09.711061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.710718    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:09.767927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.767886    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca\") pod \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") "
Oct 30 22:17:09.768100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.768058    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7\" (UniqueName: \"kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7\") pod \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") "
Oct 30 22:17:09.768154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.768143    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca\") pod \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\" (UID: \"551a4fe9-7f02-49a7-86c4-e46f7d30b534\") "
Oct 30 22:17:09.768468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.768440    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca" pod "551a4fe9-7f02-49a7-86c4-e46f7d30b534" (UID: "551a4fe9-7f02-49a7-86c4-e46f7d30b534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:09.768651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.768634    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:09.770146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.770122    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca" pod "551a4fe9-7f02-49a7-86c4-e46f7d30b534" (UID: "551a4fe9-7f02-49a7-86c4-e46f7d30b534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:09.770291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.770269    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7" pod "551a4fe9-7f02-49a7-86c4-e46f7d30b534" (UID: "551a4fe9-7f02-49a7-86c4-e46f7d30b534"). InnerVolumeSpecName "kube-api-access-vqjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:09.811301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.811202    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"]
Oct 30 22:17:09.811922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.811901    5028 cpu_manager.go:414] "containerMap: removing container" podUID="819f57b9-b878-4146-b356-5d341d7625d5" containerName="revoke-keys"
Oct 30 22:17:09.811922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.811923    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819f57b9-b878-4146-b356-5d341d7625d5" containerName="revoke-keys"
Oct 30 22:17:09.812099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.811948    5028 cpu_manager.go:414] "containerMap: removing container" podUID="551a4fe9-7f02-49a7-86c4-e46f7d30b534" containerName="revoke-keys"
Oct 30 22:17:09.812099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.811955    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551a4fe9-7f02-49a7-86c4-e46f7d30b534" containerName="revoke-keys"
Oct 30 22:17:09.812206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.812114    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="819f57b9-b878-4146-b356-5d341d7625d5" containerName="revoke-keys"
Oct 30 22:17:09.812206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.812143    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="551a4fe9-7f02-49a7-86c4-e46f7d30b534" containerName="revoke-keys"
Oct 30 22:17:09.815778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.815758    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.851885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.849243    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"]
Oct 30 22:17:09.869566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.869531    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.869788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.869681    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcd9\" (UniqueName: \"kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.869788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.869749    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.869925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.869894    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551a4fe9-7f02-49a7-86c4-e46f7d30b534-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:09.869925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.869907    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqjv7\" (UniqueName: \"kubernetes.io/projected/551a4fe9-7f02-49a7-86c4-e46f7d30b534-kube-api-access-vqjv7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:09.971190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.971149    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.971403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.971252    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.971403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.971337    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcd9\" (UniqueName: \"kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.971904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.971873    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:09.974024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:09.973998    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:10.033845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.033808    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcd9\" (UniqueName: \"kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9\") pod \"maas-api-revoke-sub-0c852b97-qwzdj\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:10.134400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.134302    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:10.485381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.485341    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:10.510630 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:10.510588    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2f278c_b7a2_43c1_8d8b_0ecb8d2d670e.slice/crio-470e3dafbf72d636ebedff5167b6cbab9c6eec0b784814121685f2152a07db5a WatchSource:0}: Error finding container 470e3dafbf72d636ebedff5167b6cbab9c6eec0b784814121685f2152a07db5a: Status 404 returned error can't find the container with id 470e3dafbf72d636ebedff5167b6cbab9c6eec0b784814121685f2152a07db5a
Oct 30 22:17:10.510799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.510773    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"]
Oct 30 22:17:10.558413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.558381    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj" event={"ID":"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e","Type":"ContainerStarted","Data":"470e3dafbf72d636ebedff5167b6cbab9c6eec0b784814121685f2152a07db5a"}
Oct 30 22:17:10.560392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.560362    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss" event={"ID":"551a4fe9-7f02-49a7-86c4-e46f7d30b534","Type":"ContainerDied","Data":"a19e0da14a2adafbe0839dcebea9c7cbaa28d16dd7b1d6c256dc6cc744548667"}
Oct 30 22:17:10.560525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.560474    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19e0da14a2adafbe0839dcebea9c7cbaa28d16dd7b1d6c256dc6cc744548667"
Oct 30 22:17:10.560588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:10.560557    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"
Oct 30 22:17:11.569213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:11.569173    5028 generic.go:356] "Generic (PLEG): container finished" podID="ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" containerID="a4132f192d7fd1df708c1deb165807ddc527ca7d6b485c609c7eccfa329bab68" exitCode=0
Oct 30 22:17:11.569636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:11.569257    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj" event={"ID":"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e","Type":"ContainerDied","Data":"a4132f192d7fd1df708c1deb165807ddc527ca7d6b485c609c7eccfa329bab68"}
Oct 30 22:17:12.900709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:12.897797    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:13.018099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.018062    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca\") pod \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") "
Oct 30 22:17:13.018411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.018388    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9\" (UniqueName: \"kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9\") pod \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") "
Oct 30 22:17:13.018625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.018602    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca\") pod \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\" (UID: \"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e\") "
Oct 30 22:17:13.020251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.020223    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca" pod "ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" (UID: "ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:13.083712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.079928    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9" pod "ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" (UID: "ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e"). InnerVolumeSpecName "kube-api-access-grcd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:13.083712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.080399    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca" pod "ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" (UID: "ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:13.128771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.120575    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.128771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.120620    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.128771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.120637    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grcd9\" (UniqueName: \"kubernetes.io/projected/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e-kube-api-access-grcd9\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:13.618720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.617319    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj" event={"ID":"ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e","Type":"ContainerDied","Data":"470e3dafbf72d636ebedff5167b6cbab9c6eec0b784814121685f2152a07db5a"}
Oct 30 22:17:13.618720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.617350    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"
Oct 30 22:17:13.618720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:13.617417    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="470e3dafbf72d636ebedff5167b6cbab9c6eec0b784814121685f2152a07db5a"
Oct 30 22:17:14.487346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.487296    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"]
Oct 30 22:17:14.488243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.488217    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" containerName="revoke-keys"
Oct 30 22:17:14.488383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.488244    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" containerName="revoke-keys"
Oct 30 22:17:14.494716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.494537    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" containerName="revoke-keys"
Oct 30 22:17:14.501771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.500394    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.544849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.544809    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"]
Oct 30 22:17:14.552192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.551577    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:14.564670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.564617    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.564890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.564731    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rc2\" (UniqueName: \"kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.564890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.564786    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.586318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.586286    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:17:14.586550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.586346    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:17:14.598192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.598163    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:17:14.665543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.665507    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.665789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.665563    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rc2\" (UniqueName: \"kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.665789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.665600    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.666466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.666437    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.668292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.668255    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.725310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.725275    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rc2\" (UniqueName: \"kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2\") pod \"maas-api-revoke-sub-78a71015-n7qmd\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:14.818804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:14.818676    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:15.207348 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:15.207316    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c117f7e_8536_4c56_b21c_5f166ed2383a.slice/crio-700394b50ca0be9928dd0560caa2c7664526553640c05d9d485be6e9c239b00d WatchSource:0}: Error finding container 700394b50ca0be9928dd0560caa2c7664526553640c05d9d485be6e9c239b00d: Status 404 returned error can't find the container with id 700394b50ca0be9928dd0560caa2c7664526553640c05d9d485be6e9c239b00d
Oct 30 22:17:15.210179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:15.210100    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"]
Oct 30 22:17:15.414847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:15.414806    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:15.639711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:15.638786    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd" event={"ID":"6c117f7e-8536-4c56-b21c-5f166ed2383a","Type":"ContainerStarted","Data":"d66ccfc4281ceae20093837b5db17bcbdf4e6154c67742f9ee0017468ad56f1c"}
Oct 30 22:17:15.639711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:15.638861    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd" event={"ID":"6c117f7e-8536-4c56-b21c-5f166ed2383a","Type":"ContainerStarted","Data":"700394b50ca0be9928dd0560caa2c7664526553640c05d9d485be6e9c239b00d"}
Oct 30 22:17:15.755357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:15.755299    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd" podStartSLOduration=1.755284538 podStartE2EDuration="1.755284538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:15.752793836 +0000 UTC m=+966.602306809" watchObservedRunningTime="2026-10-30 22:17:15.755284538 +0000 UTC m=+966.604797597"
Oct 30 22:17:16.649647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:16.649606    5028 generic.go:356] "Generic (PLEG): container finished" podID="6c117f7e-8536-4c56-b21c-5f166ed2383a" containerID="d66ccfc4281ceae20093837b5db17bcbdf4e6154c67742f9ee0017468ad56f1c" exitCode=0
Oct 30 22:17:16.650104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:16.649700    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd" event={"ID":"6c117f7e-8536-4c56-b21c-5f166ed2383a","Type":"ContainerDied","Data":"d66ccfc4281ceae20093837b5db17bcbdf4e6154c67742f9ee0017468ad56f1c"}
Oct 30 22:17:17.145733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.145671    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"]
Oct 30 22:17:17.150960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.150933    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.204104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.204070    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"]
Oct 30 22:17:17.300043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.300008    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.300241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.300153    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.300312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.300239    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55w6\" (UniqueName: \"kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.401538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.401444    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.401538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.401488    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55w6\" (UniqueName: \"kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.401819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.401564    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.402237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.402216    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.404014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.403991    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.460886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.460850    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55w6\" (UniqueName: \"kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6\") pod \"maas-api-revoke-sub-1672ea5e-gtznt\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.466207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.466177    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:17.813159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.813129    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:17.846498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.846465    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"]
Oct 30 22:17:17.847167 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:17.847140    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720ff68f_55a1_44b7_9b5f_028180c2ff4e.slice/crio-6522a5789702c3c35138fbfcdba3bdf6b2050c860565db8a005bd608f2c3c824 WatchSource:0}: Error finding container 6522a5789702c3c35138fbfcdba3bdf6b2050c860565db8a005bd608f2c3c824: Status 404 returned error can't find the container with id 6522a5789702c3c35138fbfcdba3bdf6b2050c860565db8a005bd608f2c3c824
Oct 30 22:17:17.912819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.912733    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca\") pod \"6c117f7e-8536-4c56-b21c-5f166ed2383a\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") "
Oct 30 22:17:17.912819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.912778    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca\") pod \"6c117f7e-8536-4c56-b21c-5f166ed2383a\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") "
Oct 30 22:17:17.913060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.912833    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2\" (UniqueName: \"kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2\") pod \"6c117f7e-8536-4c56-b21c-5f166ed2383a\" (UID: \"6c117f7e-8536-4c56-b21c-5f166ed2383a\") "
Oct 30 22:17:17.913211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.913167    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca" pod "6c117f7e-8536-4c56-b21c-5f166ed2383a" (UID: "6c117f7e-8536-4c56-b21c-5f166ed2383a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:17.913410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.913389    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.915026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.915001    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca" pod "6c117f7e-8536-4c56-b21c-5f166ed2383a" (UID: "6c117f7e-8536-4c56-b21c-5f166ed2383a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:17.915119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:17.915085    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2" pod "6c117f7e-8536-4c56-b21c-5f166ed2383a" (UID: "6c117f7e-8536-4c56-b21c-5f166ed2383a"). InnerVolumeSpecName "kube-api-access-h2rc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:18.014235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.014191    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rc2\" (UniqueName: \"kubernetes.io/projected/6c117f7e-8536-4c56-b21c-5f166ed2383a-kube-api-access-h2rc2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:18.014235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.014224    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c117f7e-8536-4c56-b21c-5f166ed2383a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:18.668669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.668514    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt" event={"ID":"720ff68f-55a1-44b7-9b5f-028180c2ff4e","Type":"ContainerStarted","Data":"bf3c293dbe01276f7aced38003884f0ef881a43a690f84afa7cc2cbc5cd7573e"}
Oct 30 22:17:18.668669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.668578    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt" event={"ID":"720ff68f-55a1-44b7-9b5f-028180c2ff4e","Type":"ContainerStarted","Data":"6522a5789702c3c35138fbfcdba3bdf6b2050c860565db8a005bd608f2c3c824"}
Oct 30 22:17:18.670473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.670444    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"
Oct 30 22:17:18.670620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.670489    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd" event={"ID":"6c117f7e-8536-4c56-b21c-5f166ed2383a","Type":"ContainerDied","Data":"700394b50ca0be9928dd0560caa2c7664526553640c05d9d485be6e9c239b00d"}
Oct 30 22:17:18.670620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.670535    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="700394b50ca0be9928dd0560caa2c7664526553640c05d9d485be6e9c239b00d"
Oct 30 22:17:18.787247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:18.787194    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt" podStartSLOduration=1.787178456 podStartE2EDuration="1.787178456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:18.783576542 +0000 UTC m=+969.633089515" watchObservedRunningTime="2026-10-30 22:17:18.787178456 +0000 UTC m=+969.636691429"
Oct 30 22:17:19.537203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.537159    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"]
Oct 30 22:17:19.537992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.537969    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6c117f7e-8536-4c56-b21c-5f166ed2383a" containerName="revoke-keys"
Oct 30 22:17:19.537992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.537992    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c117f7e-8536-4c56-b21c-5f166ed2383a" containerName="revoke-keys"
Oct 30 22:17:19.538179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.538165    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c117f7e-8536-4c56-b21c-5f166ed2383a" containerName="revoke-keys"
Oct 30 22:17:19.541772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.541743    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.597798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.597758    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"]
Oct 30 22:17:19.632063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.632028    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.632291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.632078    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.632291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.632177    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sbf\" (UniqueName: \"kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.679154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.679119    5028 generic.go:356] "Generic (PLEG): container finished" podID="720ff68f-55a1-44b7-9b5f-028180c2ff4e" containerID="bf3c293dbe01276f7aced38003884f0ef881a43a690f84afa7cc2cbc5cd7573e" exitCode=0
Oct 30 22:17:19.679322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.679165    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt" event={"ID":"720ff68f-55a1-44b7-9b5f-028180c2ff4e","Type":"ContainerDied","Data":"bf3c293dbe01276f7aced38003884f0ef881a43a690f84afa7cc2cbc5cd7573e"}
Oct 30 22:17:19.732924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.732884    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sbf\" (UniqueName: \"kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.733133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.733070    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.733133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.733126    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.733761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.733728    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.735516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.735493    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.792351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.792279    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sbf\" (UniqueName: \"kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf\") pod \"maas-api-revoke-sub-5299232c-5js65\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:19.855770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:19.855732    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:20.229527 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:20.229491    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fb77e5_0f21_4a94_a12b_fea8fbbc7558.slice/crio-73cf729679d98f42c128afabe63196c876325263e1dcb7217985464315b165a3 WatchSource:0}: Error finding container 73cf729679d98f42c128afabe63196c876325263e1dcb7217985464315b165a3: Status 404 returned error can't find the container with id 73cf729679d98f42c128afabe63196c876325263e1dcb7217985464315b165a3
Oct 30 22:17:20.229655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.229583    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"]
Oct 30 22:17:20.434209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.434174    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.695147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.695106    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65" event={"ID":"58fb77e5-0f21-4a94-a12b-fea8fbbc7558","Type":"ContainerStarted","Data":"a96117ad2321852d4b148cfc2192fd6c958373bd9b64b01c58fb21eeffc17b94"}
Oct 30 22:17:20.695640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.695186    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65" event={"ID":"58fb77e5-0f21-4a94-a12b-fea8fbbc7558","Type":"ContainerStarted","Data":"73cf729679d98f42c128afabe63196c876325263e1dcb7217985464315b165a3"}
Oct 30 22:17:20.818249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.818196    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65" podStartSLOduration=1.818182465 podStartE2EDuration="1.818182465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:20.814910346 +0000 UTC m=+971.664423321" watchObservedRunningTime="2026-10-30 22:17:20.818182465 +0000 UTC m=+971.667695438"
Oct 30 22:17:20.842091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.842065    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:20.968652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.968590    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca\") pod \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") "
Oct 30 22:17:20.968877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.968663    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca\") pod \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") "
Oct 30 22:17:20.968877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.968833    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6\" (UniqueName: \"kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6\") pod \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\" (UID: \"720ff68f-55a1-44b7-9b5f-028180c2ff4e\") "
Oct 30 22:17:20.969641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.969616    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca" pod "720ff68f-55a1-44b7-9b5f-028180c2ff4e" (UID: "720ff68f-55a1-44b7-9b5f-028180c2ff4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:20.975763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.971782    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6" pod "720ff68f-55a1-44b7-9b5f-028180c2ff4e" (UID: "720ff68f-55a1-44b7-9b5f-028180c2ff4e"). InnerVolumeSpecName "kube-api-access-c55w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:20.975763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:20.971893    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca" pod "720ff68f-55a1-44b7-9b5f-028180c2ff4e" (UID: "720ff68f-55a1-44b7-9b5f-028180c2ff4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:21.070573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.070492    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c55w6\" (UniqueName: \"kubernetes.io/projected/720ff68f-55a1-44b7-9b5f-028180c2ff4e-kube-api-access-c55w6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.070573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.070524    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.070573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.070534    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720ff68f-55a1-44b7-9b5f-028180c2ff4e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:21.702814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.702782    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"
Oct 30 22:17:21.703293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.702781    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt" event={"ID":"720ff68f-55a1-44b7-9b5f-028180c2ff4e","Type":"ContainerDied","Data":"6522a5789702c3c35138fbfcdba3bdf6b2050c860565db8a005bd608f2c3c824"}
Oct 30 22:17:21.703293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.702895    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6522a5789702c3c35138fbfcdba3bdf6b2050c860565db8a005bd608f2c3c824"
Oct 30 22:17:21.704464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.704440    5028 generic.go:356] "Generic (PLEG): container finished" podID="58fb77e5-0f21-4a94-a12b-fea8fbbc7558" containerID="a96117ad2321852d4b148cfc2192fd6c958373bd9b64b01c58fb21eeffc17b94" exitCode=0
Oct 30 22:17:21.704581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:21.704525    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65" event={"ID":"58fb77e5-0f21-4a94-a12b-fea8fbbc7558","Type":"ContainerDied","Data":"a96117ad2321852d4b148cfc2192fd6c958373bd9b64b01c58fb21eeffc17b94"}
Oct 30 22:17:22.899606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.899579    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:22.990536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.990436    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf\" (UniqueName: \"kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf\") pod \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") "
Oct 30 22:17:22.990536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.990506    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca\") pod \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") "
Oct 30 22:17:22.990829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.990673    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca\") pod \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\" (UID: \"58fb77e5-0f21-4a94-a12b-fea8fbbc7558\") "
Oct 30 22:17:22.991615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.991590    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca" pod "58fb77e5-0f21-4a94-a12b-fea8fbbc7558" (UID: "58fb77e5-0f21-4a94-a12b-fea8fbbc7558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:22.993928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.993902    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca" pod "58fb77e5-0f21-4a94-a12b-fea8fbbc7558" (UID: "58fb77e5-0f21-4a94-a12b-fea8fbbc7558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:22.994040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:22.993997    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf" pod "58fb77e5-0f21-4a94-a12b-fea8fbbc7558" (UID: "58fb77e5-0f21-4a94-a12b-fea8fbbc7558"). InnerVolumeSpecName "kube-api-access-h6sbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:23.091768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.091734    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6sbf\" (UniqueName: \"kubernetes.io/projected/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-kube-api-access-h6sbf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:23.091768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.091766    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:23.091967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.091777    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58fb77e5-0f21-4a94-a12b-fea8fbbc7558-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:23.445661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.445618    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"]
Oct 30 22:17:23.446199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.446183    5028 cpu_manager.go:414] "containerMap: removing container" podUID="720ff68f-55a1-44b7-9b5f-028180c2ff4e" containerName="revoke-keys"
Oct 30 22:17:23.446199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.446198    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720ff68f-55a1-44b7-9b5f-028180c2ff4e" containerName="revoke-keys"
Oct 30 22:17:23.446368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.446223    5028 cpu_manager.go:414] "containerMap: removing container" podUID="58fb77e5-0f21-4a94-a12b-fea8fbbc7558" containerName="revoke-keys"
Oct 30 22:17:23.446368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.446228    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fb77e5-0f21-4a94-a12b-fea8fbbc7558" containerName="revoke-keys"
Oct 30 22:17:23.446368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.446343    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fb77e5-0f21-4a94-a12b-fea8fbbc7558" containerName="revoke-keys"
Oct 30 22:17:23.446368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.446356    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="720ff68f-55a1-44b7-9b5f-028180c2ff4e" containerName="revoke-keys"
Oct 30 22:17:23.450149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.450126    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.495920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.495879    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.496113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.496019    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rps5k\" (UniqueName: \"kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.496113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.496077    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.503618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.503579    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"]
Oct 30 22:17:23.596772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.596642    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rps5k\" (UniqueName: \"kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.596772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.596765    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.597025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.596862    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.597543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.597515    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.599192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.599169    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.655942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.655905    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rps5k\" (UniqueName: \"kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k\") pod \"maas-api-revoke-sub-41b253be-t678c\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.724223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.724187    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65" event={"ID":"58fb77e5-0f21-4a94-a12b-fea8fbbc7558","Type":"ContainerDied","Data":"73cf729679d98f42c128afabe63196c876325263e1dcb7217985464315b165a3"}
Oct 30 22:17:23.724395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.724237    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73cf729679d98f42c128afabe63196c876325263e1dcb7217985464315b165a3"
Oct 30 22:17:23.724395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.724239    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"
Oct 30 22:17:23.771789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.771755    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:23.864176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.864095    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"]
Oct 30 22:17:23.869374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.869359    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:23.899853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.899821    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkzl\" (UniqueName: \"kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:23.900256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.899993    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:23.900256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.900044    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:23.957407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:23.957362    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"]
Oct 30 22:17:24.000872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.000835    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkzl\" (UniqueName: \"kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.001057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.000973    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.001057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.001012    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.001676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.001648    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.003407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.003386    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.059379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.059347    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkzl\" (UniqueName: \"kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl\") pod \"maas-api-revoke-sub-314ad3f9-kf6ml\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.147586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.147557    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"]
Oct 30 22:17:24.147919 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:24.147897    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0fa524_2bbf_4798_a6d2_14bbf43eded1.slice/crio-4713e7dc183c391d93c7b9b685210e41ba77a230bcb4597cae88bdb876670433 WatchSource:0}: Error finding container 4713e7dc183c391d93c7b9b685210e41ba77a230bcb4597cae88bdb876670433: Status 404 returned error can't find the container with id 4713e7dc183c391d93c7b9b685210e41ba77a230bcb4597cae88bdb876670433
Oct 30 22:17:24.201997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.201967    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:24.577004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.576972    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"]
Oct 30 22:17:24.577189 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:24.577168    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode407743a_c859_4dd8_96c0_1757a4d74c37.slice/crio-f70a73356e6d4cad4bf207f77871b58ee02bcc720e8a647a31ae0a9318b97eec WatchSource:0}: Error finding container f70a73356e6d4cad4bf207f77871b58ee02bcc720e8a647a31ae0a9318b97eec: Status 404 returned error can't find the container with id f70a73356e6d4cad4bf207f77871b58ee02bcc720e8a647a31ae0a9318b97eec
Oct 30 22:17:24.731614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.731578    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml" event={"ID":"e407743a-c859-4dd8-96c0-1757a4d74c37","Type":"ContainerStarted","Data":"f70a73356e6d4cad4bf207f77871b58ee02bcc720e8a647a31ae0a9318b97eec"}
Oct 30 22:17:24.733193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.733162    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c" event={"ID":"2d0fa524-2bbf-4798-a6d2-14bbf43eded1","Type":"ContainerStarted","Data":"cb145b6592bf11c3314e38a95b36f4cda3cd02e99708bad92e673890d047b429"}
Oct 30 22:17:24.733340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.733211    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c" event={"ID":"2d0fa524-2bbf-4798-a6d2-14bbf43eded1","Type":"ContainerStarted","Data":"4713e7dc183c391d93c7b9b685210e41ba77a230bcb4597cae88bdb876670433"}
Oct 30 22:17:24.847859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:24.847797    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c" podStartSLOduration=1.847781677 podStartE2EDuration="1.847781677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:24.844804317 +0000 UTC m=+975.694317291" watchObservedRunningTime="2026-10-30 22:17:24.847781677 +0000 UTC m=+975.697294649"
Oct 30 22:17:25.200927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:25.200881    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:25.757864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:25.756556    5028 generic.go:356] "Generic (PLEG): container finished" podID="e407743a-c859-4dd8-96c0-1757a4d74c37" containerID="204625d9392954d9decd5e244a765b84837d8109d8f88291626bc7f8f0965d82" exitCode=0
Oct 30 22:17:25.757864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:25.756647    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml" event={"ID":"e407743a-c859-4dd8-96c0-1757a4d74c37","Type":"ContainerDied","Data":"204625d9392954d9decd5e244a765b84837d8109d8f88291626bc7f8f0965d82"}
Oct 30 22:17:25.761428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:25.761399    5028 generic.go:356] "Generic (PLEG): container finished" podID="2d0fa524-2bbf-4798-a6d2-14bbf43eded1" containerID="cb145b6592bf11c3314e38a95b36f4cda3cd02e99708bad92e673890d047b429" exitCode=0
Oct 30 22:17:25.761716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:25.761603    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c" event={"ID":"2d0fa524-2bbf-4798-a6d2-14bbf43eded1","Type":"ContainerDied","Data":"cb145b6592bf11c3314e38a95b36f4cda3cd02e99708bad92e673890d047b429"}
Oct 30 22:17:26.532087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.532050    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"]
Oct 30 22:17:26.536276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.536257    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.640751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.640708    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.640946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.640761    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.640946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.640859    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.640946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.640891    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.641133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.640995    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.641133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.641075    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.641133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.641098    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45z7\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.641297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.641201    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.641297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.641273    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.642848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.642823    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"]
Oct 30 22:17:26.741909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.741868    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.741961    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.741994    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742024    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742065    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742093    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742123    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742182    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742210    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45z7\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742422    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742497    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742570    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742771    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.742932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.742913    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.744472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.744445    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.744717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.744681    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.803167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.803087    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.803577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.803555    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45z7\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdcn8\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.852596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.852277    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:26.921454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.921131    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:26.959570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:26.959543    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:27.048002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.047971    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca\") pod \"e407743a-c859-4dd8-96c0-1757a4d74c37\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") "
Oct 30 22:17:27.048198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048103    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl\" (UniqueName: \"kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl\") pod \"e407743a-c859-4dd8-96c0-1757a4d74c37\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") "
Oct 30 22:17:27.048198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048132    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k\" (UniqueName: \"kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k\") pod \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") "
Oct 30 22:17:27.048198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048180    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca\") pod \"e407743a-c859-4dd8-96c0-1757a4d74c37\" (UID: \"e407743a-c859-4dd8-96c0-1757a4d74c37\") "
Oct 30 22:17:27.048381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048248    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca\") pod \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") "
Oct 30 22:17:27.048440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048414    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca" pod "e407743a-c859-4dd8-96c0-1757a4d74c37" (UID: "e407743a-c859-4dd8-96c0-1757a4d74c37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:27.048491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048474    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca\") pod \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\" (UID: \"2d0fa524-2bbf-4798-a6d2-14bbf43eded1\") "
Oct 30 22:17:27.048942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.048911    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca" pod "2d0fa524-2bbf-4798-a6d2-14bbf43eded1" (UID: "2d0fa524-2bbf-4798-a6d2-14bbf43eded1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:27.049107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.049089    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.049167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.049111    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.051000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.050972    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl" pod "e407743a-c859-4dd8-96c0-1757a4d74c37" (UID: "e407743a-c859-4dd8-96c0-1757a4d74c37"). InnerVolumeSpecName "kube-api-access-zkkzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:27.051129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.051017    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca" pod "e407743a-c859-4dd8-96c0-1757a4d74c37" (UID: "e407743a-c859-4dd8-96c0-1757a4d74c37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:27.051129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.051023    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca" pod "2d0fa524-2bbf-4798-a6d2-14bbf43eded1" (UID: "2d0fa524-2bbf-4798-a6d2-14bbf43eded1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:27.051129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.051063    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k" pod "2d0fa524-2bbf-4798-a6d2-14bbf43eded1" (UID: "2d0fa524-2bbf-4798-a6d2-14bbf43eded1"). InnerVolumeSpecName "kube-api-access-rps5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:27.150490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.150318    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.150490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.150366    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkzl\" (UniqueName: \"kubernetes.io/projected/e407743a-c859-4dd8-96c0-1757a4d74c37-kube-api-access-zkkzl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.150490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.150381    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rps5k\" (UniqueName: \"kubernetes.io/projected/2d0fa524-2bbf-4798-a6d2-14bbf43eded1-kube-api-access-rps5k\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.150490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.150396    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e407743a-c859-4dd8-96c0-1757a4d74c37-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.299491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.299462    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"]
Oct 30 22:17:27.301348 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:27.301302    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbb79d5_a4a8_40c9_b12f_7f1c3ff9e56b.slice/crio-1bee575138aa0bb53305f8062785b9d2abf6c043ded6c9e88c1a7f8143dec8bf WatchSource:0}: Error finding container 1bee575138aa0bb53305f8062785b9d2abf6c043ded6c9e88c1a7f8143dec8bf: Status 404 returned error can't find the container with id 1bee575138aa0bb53305f8062785b9d2abf6c043ded6c9e88c1a7f8143dec8bf
Oct 30 22:17:27.304181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.304148    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:27.304308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.304231    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:27.304308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.304283    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:27.779131 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.779088    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" event={"ID":"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b","Type":"ContainerStarted","Data":"a8aafca25e0a7ef81b53e6793456244aa30c49ea07b7fcc2a300997fc7da1ed0"}
Oct 30 22:17:27.779971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.779153    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" event={"ID":"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b","Type":"ContainerStarted","Data":"1bee575138aa0bb53305f8062785b9d2abf6c043ded6c9e88c1a7f8143dec8bf"}
Oct 30 22:17:27.781951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.781927    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"
Oct 30 22:17:27.782081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.781922    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml" event={"ID":"e407743a-c859-4dd8-96c0-1757a4d74c37","Type":"ContainerDied","Data":"f70a73356e6d4cad4bf207f77871b58ee02bcc720e8a647a31ae0a9318b97eec"}
Oct 30 22:17:27.782134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.782087    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f70a73356e6d4cad4bf207f77871b58ee02bcc720e8a647a31ae0a9318b97eec"
Oct 30 22:17:27.784418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.784393    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c" event={"ID":"2d0fa524-2bbf-4798-a6d2-14bbf43eded1","Type":"ContainerDied","Data":"4713e7dc183c391d93c7b9b685210e41ba77a230bcb4597cae88bdb876670433"}
Oct 30 22:17:27.784584 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.784564    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4713e7dc183c391d93c7b9b685210e41ba77a230bcb4597cae88bdb876670433"
Oct 30 22:17:27.785050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.785033    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"
Oct 30 22:17:27.854712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.852765    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:27.939710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:27.939132    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podStartSLOduration=1.939109011 podStartE2EDuration="1.939109011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:27.895362677 +0000 UTC m=+978.744875649" watchObservedRunningTime="2026-10-30 22:17:27.939109011 +0000 UTC m=+978.788621985"
Oct 30 22:17:28.097788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.097707    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.175947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.175912    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"]
Oct 30 22:17:28.176806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.176783    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2d0fa524-2bbf-4798-a6d2-14bbf43eded1" containerName="revoke-keys"
Oct 30 22:17:28.176806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.176804    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0fa524-2bbf-4798-a6d2-14bbf43eded1" containerName="revoke-keys"
Oct 30 22:17:28.176806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.176819    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e407743a-c859-4dd8-96c0-1757a4d74c37" containerName="revoke-keys"
Oct 30 22:17:28.177061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.176826    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e407743a-c859-4dd8-96c0-1757a4d74c37" containerName="revoke-keys"
Oct 30 22:17:28.177061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.176940    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0fa524-2bbf-4798-a6d2-14bbf43eded1" containerName="revoke-keys"
Oct 30 22:17:28.177061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.176949    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e407743a-c859-4dd8-96c0-1757a4d74c37" containerName="revoke-keys"
Oct 30 22:17:28.182047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.182024    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.200363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.200332    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"]
Oct 30 22:17:28.248407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.248048    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:17:28.248407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.248207    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:17:28.248407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.248295    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:17:28.248407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.248076    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:28.275680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.275647    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9mg\" (UniqueName: \"kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.275892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.275720    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.275892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.275858    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.377511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.377318    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.377511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.377425    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9mg\" (UniqueName: \"kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.377511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.377469    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.378291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.378261    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.380069 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.380045    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.458576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.458532    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9mg\" (UniqueName: \"kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg\") pod \"maas-api-revoke-sub-1891ceae-752sh\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.476893 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:17:28.476854    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:17:28.496081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.496049    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:28.855192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.855148    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:28.855616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.855224    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:28.907283 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:28.907239    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d44fe5_c600_4c0c_8012_1d40c0c44346.slice/crio-a059b9d27bf5250ba4df28cb9da3de89ffb8d565b2f14e45ad354897acd407af WatchSource:0}: Error finding container a059b9d27bf5250ba4df28cb9da3de89ffb8d565b2f14e45ad354897acd407af: Status 404 returned error can't find the container with id a059b9d27bf5250ba4df28cb9da3de89ffb8d565b2f14e45ad354897acd407af
Oct 30 22:17:28.907789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:28.907766    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"]
Oct 30 22:17:29.801777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:29.801725    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh" event={"ID":"f9d44fe5-c600-4c0c-8012-1d40c0c44346","Type":"ContainerStarted","Data":"ddbc67f4edd49c1fa274270f056b7b2cd49bdfe78cb09f17b4fe52a89eb8059c"}
Oct 30 22:17:29.801980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:29.801796    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh" event={"ID":"f9d44fe5-c600-4c0c-8012-1d40c0c44346","Type":"ContainerStarted","Data":"a059b9d27bf5250ba4df28cb9da3de89ffb8d565b2f14e45ad354897acd407af"}
Oct 30 22:17:29.852880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:29.852842    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:29.853081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:29.852899    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:29.916504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:29.916449    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh" podStartSLOduration=1.916434192 podStartE2EDuration="1.916434192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:29.912553317 +0000 UTC m=+980.762066291" watchObservedRunningTime="2026-10-30 22:17:29.916434192 +0000 UTC m=+980.765947164"
Oct 30 22:17:30.063978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:30.063894    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:30.810442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:30.810401    5028 generic.go:356] "Generic (PLEG): container finished" podID="f9d44fe5-c600-4c0c-8012-1d40c0c44346" containerID="ddbc67f4edd49c1fa274270f056b7b2cd49bdfe78cb09f17b4fe52a89eb8059c" exitCode=0
Oct 30 22:17:30.810646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:30.810491    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh" event={"ID":"f9d44fe5-c600-4c0c-8012-1d40c0c44346","Type":"ContainerDied","Data":"ddbc67f4edd49c1fa274270f056b7b2cd49bdfe78cb09f17b4fe52a89eb8059c"}
Oct 30 22:17:30.853708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:30.853654    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:30.853896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:30.853738    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:31.853442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:31.853400    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:31.853894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:31.853480    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:32.096000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.095931    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:32.259159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.259106    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca\") pod \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") "
Oct 30 22:17:32.259347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.259241    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg\" (UniqueName: \"kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg\") pod \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") "
Oct 30 22:17:32.259347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.259332    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca\") pod \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\" (UID: \"f9d44fe5-c600-4c0c-8012-1d40c0c44346\") "
Oct 30 22:17:32.260164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.260120    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca" pod "f9d44fe5-c600-4c0c-8012-1d40c0c44346" (UID: "f9d44fe5-c600-4c0c-8012-1d40c0c44346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:32.264184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.264153    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca" pod "f9d44fe5-c600-4c0c-8012-1d40c0c44346" (UID: "f9d44fe5-c600-4c0c-8012-1d40c0c44346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:32.264540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.264519    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg" pod "f9d44fe5-c600-4c0c-8012-1d40c0c44346" (UID: "f9d44fe5-c600-4c0c-8012-1d40c0c44346"). InnerVolumeSpecName "kube-api-access-qt9mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:32.367438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.361503    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9mg\" (UniqueName: \"kubernetes.io/projected/f9d44fe5-c600-4c0c-8012-1d40c0c44346-kube-api-access-qt9mg\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:32.367438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.361537    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:32.367438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.361553    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d44fe5-c600-4c0c-8012-1d40c0c44346-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:32.785863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.785828    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"]
Oct 30 22:17:32.786440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.786427    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f9d44fe5-c600-4c0c-8012-1d40c0c44346" containerName="revoke-keys"
Oct 30 22:17:32.786481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.786441    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d44fe5-c600-4c0c-8012-1d40c0c44346" containerName="revoke-keys"
Oct 30 22:17:32.786527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.786519    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d44fe5-c600-4c0c-8012-1d40c0c44346" containerName="revoke-keys"
Oct 30 22:17:32.790166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.790145    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.854478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.854360    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:32.854478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.854418    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:32.859261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.859230    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"
Oct 30 22:17:32.859481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.859327    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh" event={"ID":"f9d44fe5-c600-4c0c-8012-1d40c0c44346","Type":"ContainerDied","Data":"a059b9d27bf5250ba4df28cb9da3de89ffb8d565b2f14e45ad354897acd407af"}
Oct 30 22:17:32.859481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.859380    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a059b9d27bf5250ba4df28cb9da3de89ffb8d565b2f14e45ad354897acd407af"
Oct 30 22:17:32.877414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.872879    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.877414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.873367    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5285\" (UniqueName: \"kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.877414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.873419    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.879985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.879952    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"]
Oct 30 22:17:32.974860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.974820    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5285\" (UniqueName: \"kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.975045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.974880    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.975111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.975061    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.975658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.975620    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:32.977540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:32.977516    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:33.058429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.058347    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5285\" (UniqueName: \"kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285\") pod \"maas-api-revoke-sub-f3577d5d-lwwld\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:33.061115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.061083    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"]
Oct 30 22:17:33.066937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.066914    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.097611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.097576    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"]
Oct 30 22:17:33.104843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.104812    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:33.177510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177477    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64bw\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.177696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177608    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.177696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177673    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.177849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177705    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.177849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177762    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.177849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177819    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.178016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177878    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.178016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177904    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.178016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.177945    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279119    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279169    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279191    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279209    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279250    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279275    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279296    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279331    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279369    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64bw\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.279521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.279465    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.280044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.280004    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.280044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.280032    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.280281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.280218    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.280673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.280647    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.284103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.284072    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.284513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.284470    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.340647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.340526    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.343132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.343102    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64bw\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.386596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.386554    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:33.527987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.527949    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"]
Oct 30 22:17:33.529997 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:33.528953    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b085ea_e1dc_4573_b9a5_344f09d0d863.slice/crio-d9854b252a196ba05e35ef75f5ab73b5b15f306b2ee41b35d02761b26356c703 WatchSource:0}: Error finding container d9854b252a196ba05e35ef75f5ab73b5b15f306b2ee41b35d02761b26356c703: Status 404 returned error can't find the container with id d9854b252a196ba05e35ef75f5ab73b5b15f306b2ee41b35d02761b26356c703
Oct 30 22:17:33.773613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.773578    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"]
Oct 30 22:17:33.773989 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:33.773959    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5405bee9_16b0_463b_a2c9_ca83c4e629f8.slice/crio-3f4105f6a2f914efd88dacec88bd23425fe739235fb9cfc4b861fc3e05663915 WatchSource:0}: Error finding container 3f4105f6a2f914efd88dacec88bd23425fe739235fb9cfc4b861fc3e05663915: Status 404 returned error can't find the container with id 3f4105f6a2f914efd88dacec88bd23425fe739235fb9cfc4b861fc3e05663915
Oct 30 22:17:33.776421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.776387    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:33.776514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.776450    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:33.776514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.776478    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:33.853946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.853916    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:33.854130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.853966    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:33.871512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.871476    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" event={"ID":"5405bee9-16b0-463b-a2c9-ca83c4e629f8","Type":"ContainerStarted","Data":"715e19bbc6bb753d9a907e1b12f7d95cb013a416503bae99f1b42f95ca1c9970"}
Oct 30 22:17:33.871956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.871538    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" event={"ID":"5405bee9-16b0-463b-a2c9-ca83c4e629f8","Type":"ContainerStarted","Data":"3f4105f6a2f914efd88dacec88bd23425fe739235fb9cfc4b861fc3e05663915"}
Oct 30 22:17:33.873305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.873280    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld" event={"ID":"b6b085ea-e1dc-4573-b9a5-344f09d0d863","Type":"ContainerStarted","Data":"d9854b252a196ba05e35ef75f5ab73b5b15f306b2ee41b35d02761b26356c703"}
Oct 30 22:17:33.998295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:33.998241    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podStartSLOduration=1.998225459 podStartE2EDuration="1.998225459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:33.995362743 +0000 UTC m=+984.844875716" watchObservedRunningTime="2026-10-30 22:17:33.998225459 +0000 UTC m=+984.847738432"
Oct 30 22:17:34.387572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.387528    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:34.389578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.389546    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:34.389749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.389608    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:34.693911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.693880    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"]
Oct 30 22:17:34.697929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.697906    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.761392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.757775    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"]
Oct 30 22:17:34.799419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.799384    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.799594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.799481    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.799594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.799577    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlf4\" (UniqueName: \"kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.855711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.855130    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:17:34.855711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.855190    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 22:17:34.891829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.891791    5028 generic.go:356] "Generic (PLEG): container finished" podID="b6b085ea-e1dc-4573-b9a5-344f09d0d863" containerID="7db01f7831254dd631cbd866c302f4ff977988c4fb7d9d7d0039de53d6b8e9c3" exitCode=0
Oct 30 22:17:34.892230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.891830    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld" event={"ID":"b6b085ea-e1dc-4573-b9a5-344f09d0d863","Type":"ContainerDied","Data":"7db01f7831254dd631cbd866c302f4ff977988c4fb7d9d7d0039de53d6b8e9c3"}
Oct 30 22:17:34.901182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.901154    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.901357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.901229    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.901357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.901265    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlf4\" (UniqueName: \"kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.902225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.902195    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.903632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.903607    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.962943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.962793    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlf4\" (UniqueName: \"kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4\") pod \"maas-api-revoke-sub-ed82e841-lhx2r\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:34.973364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:34.973328    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:35.076717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:35.075913    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:35.387736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:35.387636    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:35.387736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:35.387706    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:35.502803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:35.502761    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"]
Oct 30 22:17:35.504551 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:35.504523    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf6eca3_c77f_45cc_b33a_25c4addcf3b6.slice/crio-5573b81af48f44c510395258b10e025dcb16ec55f7819b99334c979d466e0cec WatchSource:0}: Error finding container 5573b81af48f44c510395258b10e025dcb16ec55f7819b99334c979d466e0cec: Status 404 returned error can't find the container with id 5573b81af48f44c510395258b10e025dcb16ec55f7819b99334c979d466e0cec
Oct 30 22:17:35.904986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:35.904930    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r" event={"ID":"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6","Type":"ContainerStarted","Data":"5573b81af48f44c510395258b10e025dcb16ec55f7819b99334c979d466e0cec"}
Oct 30 22:17:36.398477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.397836    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:36.398477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.397911    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:36.412727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.407286    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:36.454739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.454075    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285\" (UniqueName: \"kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285\") pod \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") "
Oct 30 22:17:36.454739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.454171    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca\") pod \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") "
Oct 30 22:17:36.454739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.454346    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca\") pod \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\" (UID: \"b6b085ea-e1dc-4573-b9a5-344f09d0d863\") "
Oct 30 22:17:36.474402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.474359    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca" pod "b6b085ea-e1dc-4573-b9a5-344f09d0d863" (UID: "b6b085ea-e1dc-4573-b9a5-344f09d0d863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:36.485770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.474898    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca" pod "b6b085ea-e1dc-4573-b9a5-344f09d0d863" (UID: "b6b085ea-e1dc-4573-b9a5-344f09d0d863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:36.485770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.484267    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285" pod "b6b085ea-e1dc-4573-b9a5-344f09d0d863" (UID: "b6b085ea-e1dc-4573-b9a5-344f09d0d863"). InnerVolumeSpecName "kube-api-access-m5285". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:36.560767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.560624    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:36.560767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.560665    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5285\" (UniqueName: \"kubernetes.io/projected/b6b085ea-e1dc-4573-b9a5-344f09d0d863-kube-api-access-m5285\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:36.571277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.560699    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b085ea-e1dc-4573-b9a5-344f09d0d863-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:36.810123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.809107    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:36.811270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.810423    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:36.812860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.812835    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:17:36.960502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.960461    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld" event={"ID":"b6b085ea-e1dc-4573-b9a5-344f09d0d863","Type":"ContainerDied","Data":"d9854b252a196ba05e35ef75f5ab73b5b15f306b2ee41b35d02761b26356c703"}
Oct 30 22:17:36.961132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.961107    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9854b252a196ba05e35ef75f5ab73b5b15f306b2ee41b35d02761b26356c703"
Oct 30 22:17:36.961340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:36.961323    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"
Oct 30 22:17:37.020713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:37.015331    5028 generic.go:356] "Generic (PLEG): container finished" podID="0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" containerID="7ca3ea8b3518e9160c54f935f1bfbb17dbf7825023dcb31dfebfdbcc758a406f" exitCode=0
Oct 30 22:17:37.020713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:37.015569    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r" event={"ID":"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6","Type":"ContainerDied","Data":"7ca3ea8b3518e9160c54f935f1bfbb17dbf7825023dcb31dfebfdbcc758a406f"}
Oct 30 22:17:37.573786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:37.569905    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:37.573786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:37.569977    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:38.390142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.390026    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:38.390142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.390104    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:38.575793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.575419    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:38.615970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.615929    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca\") pod \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") "
Oct 30 22:17:38.616158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.615979    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca\") pod \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") "
Oct 30 22:17:38.616158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.616050    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4\" (UniqueName: \"kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4\") pod \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\" (UID: \"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6\") "
Oct 30 22:17:38.616400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.616373    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca" pod "0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" (UID: "0cf6eca3-c77f-45cc-b33a-25c4addcf3b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:38.616553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.616536    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:38.653414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.647862    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca" pod "0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" (UID: "0cf6eca3-c77f-45cc-b33a-25c4addcf3b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:38.658926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.657377    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4" pod "0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" (UID: "0cf6eca3-c77f-45cc-b33a-25c4addcf3b6"). InnerVolumeSpecName "kube-api-access-qzlf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:38.717460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.717421    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:38.717460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.717459    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzlf4\" (UniqueName: \"kubernetes.io/projected/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6-kube-api-access-qzlf4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:38.861300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.861191    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"]
Oct 30 22:17:38.862138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.862110    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" containerName="revoke-keys"
Oct 30 22:17:38.862138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.862135    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" containerName="revoke-keys"
Oct 30 22:17:38.862337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.862162    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b6b085ea-e1dc-4573-b9a5-344f09d0d863" containerName="revoke-keys"
Oct 30 22:17:38.862337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.862170    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b085ea-e1dc-4573-b9a5-344f09d0d863" containerName="revoke-keys"
Oct 30 22:17:38.862456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.862340    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b085ea-e1dc-4573-b9a5-344f09d0d863" containerName="revoke-keys"
Oct 30 22:17:38.862456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.862363    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" containerName="revoke-keys"
Oct 30 22:17:38.866123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.866100    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:38.958625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.958577    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927lj\" (UniqueName: \"kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:38.959078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.959050    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:38.959377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.959356    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:38.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:38.996145    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"]
Oct 30 22:17:39.067746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.064910    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.067746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.065027    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.067746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.065147    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927lj\" (UniqueName: \"kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.067746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.067511    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.083710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.081275    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r" event={"ID":"0cf6eca3-c77f-45cc-b33a-25c4addcf3b6","Type":"ContainerDied","Data":"5573b81af48f44c510395258b10e025dcb16ec55f7819b99334c979d466e0cec"}
Oct 30 22:17:39.083710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.081371    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5573b81af48f44c510395258b10e025dcb16ec55f7819b99334c979d466e0cec"
Oct 30 22:17:39.083710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.081464    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"
Oct 30 22:17:39.097987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.095555    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.143792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.142782    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927lj\" (UniqueName: \"kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj\") pod \"maas-api-revoke-sub-9437dd04-wxczd\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.185797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.185557    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:39.387848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.387334    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:39.387848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.387396    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:39.670743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.669791    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"]
Oct 30 22:17:39.861716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.861047    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"]
Oct 30 22:17:39.867231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.867200    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:39.872063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.872012    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:39.909294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.909047    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"]
Oct 30 22:17:39.984713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.984102    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:39.984713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.984166    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wwv\" (UniqueName: \"kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:39.984713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:39.984248    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.085875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.085833    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.086098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.086018    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.087125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.087097    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.087327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.087154    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wwv\" (UniqueName: \"kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.092387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.092365    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.093474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.093445    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd" event={"ID":"a3a24cae-60e8-4f64-8db0-296929f4cb92","Type":"ContainerStarted","Data":"b4b2b9468fc22991d5db5a602f5ce1e604ca9ae1172a30f30144265e39a1df9a"}
Oct 30 22:17:40.145652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.145616    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wwv\" (UniqueName: \"kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv\") pod \"maas-api-revoke-sub-587e4216-jdtht\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.186734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.186670    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:40.387633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.387589    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:17:40.387850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.387667    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 22:17:40.570043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:40.570013    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"]
Oct 30 22:17:40.571305 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:40.571274    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275f769a_1442_46b7_b905_b88c56205c49.slice/crio-be1b94e1e49c70d758a72782b3f1caca5ed7e28de8efb08526e50b25696671de WatchSource:0}: Error finding container be1b94e1e49c70d758a72782b3f1caca5ed7e28de8efb08526e50b25696671de: Status 404 returned error can't find the container with id be1b94e1e49c70d758a72782b3f1caca5ed7e28de8efb08526e50b25696671de
Oct 30 22:17:41.103008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.102975    5028 generic.go:356] "Generic (PLEG): container finished" podID="a3a24cae-60e8-4f64-8db0-296929f4cb92" containerID="ec5d65819ca30e8fe7c10346df57fecf9dc7a4b8976cc766a0e044b44196a8fe" exitCode=0
Oct 30 22:17:41.103507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.103072    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd" event={"ID":"a3a24cae-60e8-4f64-8db0-296929f4cb92","Type":"ContainerDied","Data":"ec5d65819ca30e8fe7c10346df57fecf9dc7a4b8976cc766a0e044b44196a8fe"}
Oct 30 22:17:41.104928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.104887    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht" event={"ID":"275f769a-1442-46b7-b905-b88c56205c49","Type":"ContainerStarted","Data":"33a0316aefc2788c86975b95dd74eb8c9a038550d2446735708dd2b5f6f9b92e"}
Oct 30 22:17:41.105067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.104946    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht" event={"ID":"275f769a-1442-46b7-b905-b88c56205c49","Type":"ContainerStarted","Data":"be1b94e1e49c70d758a72782b3f1caca5ed7e28de8efb08526e50b25696671de"}
Oct 30 22:17:41.327680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.327619    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht" podStartSLOduration=2.327605276 podStartE2EDuration="2.327605276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:41.324180588 +0000 UTC m=+992.173693583" watchObservedRunningTime="2026-10-30 22:17:41.327605276 +0000 UTC m=+992.177118332"
Oct 30 22:17:41.754626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.752910    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:41.754626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.754351    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:41.807964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:41.807932    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:17:42.113862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.113824    5028 generic.go:356] "Generic (PLEG): container finished" podID="275f769a-1442-46b7-b905-b88c56205c49" containerID="33a0316aefc2788c86975b95dd74eb8c9a038550d2446735708dd2b5f6f9b92e" exitCode=0
Oct 30 22:17:42.114297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.113889    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht" event={"ID":"275f769a-1442-46b7-b905-b88c56205c49","Type":"ContainerDied","Data":"33a0316aefc2788c86975b95dd74eb8c9a038550d2446735708dd2b5f6f9b92e"}
Oct 30 22:17:42.281430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.281400    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:42.415617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.415525    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj\" (UniqueName: \"kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj\") pod \"a3a24cae-60e8-4f64-8db0-296929f4cb92\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") "
Oct 30 22:17:42.415617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.415610    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca\") pod \"a3a24cae-60e8-4f64-8db0-296929f4cb92\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") "
Oct 30 22:17:42.415829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.415799    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca\") pod \"a3a24cae-60e8-4f64-8db0-296929f4cb92\" (UID: \"a3a24cae-60e8-4f64-8db0-296929f4cb92\") "
Oct 30 22:17:42.416160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.416128    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca" pod "a3a24cae-60e8-4f64-8db0-296929f4cb92" (UID: "a3a24cae-60e8-4f64-8db0-296929f4cb92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:42.416653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.416624    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:42.418317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.418295    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca" pod "a3a24cae-60e8-4f64-8db0-296929f4cb92" (UID: "a3a24cae-60e8-4f64-8db0-296929f4cb92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:42.418392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.418318    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj" pod "a3a24cae-60e8-4f64-8db0-296929f4cb92" (UID: "a3a24cae-60e8-4f64-8db0-296929f4cb92"). InnerVolumeSpecName "kube-api-access-927lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:42.518506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.518460    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927lj\" (UniqueName: \"kubernetes.io/projected/a3a24cae-60e8-4f64-8db0-296929f4cb92-kube-api-access-927lj\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:42.518506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:42.518504    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a24cae-60e8-4f64-8db0-296929f4cb92-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.125289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.125251    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"
Oct 30 22:17:43.125289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.125274    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd" event={"ID":"a3a24cae-60e8-4f64-8db0-296929f4cb92","Type":"ContainerDied","Data":"b4b2b9468fc22991d5db5a602f5ce1e604ca9ae1172a30f30144265e39a1df9a"}
Oct 30 22:17:43.125887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.125369    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b2b9468fc22991d5db5a602f5ce1e604ca9ae1172a30f30144265e39a1df9a"
Oct 30 22:17:43.287870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.287841    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:43.430355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.430262    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca\") pod \"275f769a-1442-46b7-b905-b88c56205c49\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") "
Oct 30 22:17:43.430520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.430435    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca\") pod \"275f769a-1442-46b7-b905-b88c56205c49\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") "
Oct 30 22:17:43.430605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.430530    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv\" (UniqueName: \"kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv\") pod \"275f769a-1442-46b7-b905-b88c56205c49\" (UID: \"275f769a-1442-46b7-b905-b88c56205c49\") "
Oct 30 22:17:43.431296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.430850    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca" pod "275f769a-1442-46b7-b905-b88c56205c49" (UID: "275f769a-1442-46b7-b905-b88c56205c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:43.431296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.431076    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f769a-1442-46b7-b905-b88c56205c49-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.433087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.433057    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv" pod "275f769a-1442-46b7-b905-b88c56205c49" (UID: "275f769a-1442-46b7-b905-b88c56205c49"). InnerVolumeSpecName "kube-api-access-r6wwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:43.433189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.433106    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca" pod "275f769a-1442-46b7-b905-b88c56205c49" (UID: "275f769a-1442-46b7-b905-b88c56205c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:43.532277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.532237    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6wwv\" (UniqueName: \"kubernetes.io/projected/275f769a-1442-46b7-b905-b88c56205c49-kube-api-access-r6wwv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.532277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:43.532271    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f769a-1442-46b7-b905-b88c56205c49-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:44.133265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:44.133230    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"
Oct 30 22:17:44.133649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:44.133273    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht" event={"ID":"275f769a-1442-46b7-b905-b88c56205c49","Type":"ContainerDied","Data":"be1b94e1e49c70d758a72782b3f1caca5ed7e28de8efb08526e50b25696671de"}
Oct 30 22:17:44.133649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:44.133317    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be1b94e1e49c70d758a72782b3f1caca5ed7e28de8efb08526e50b25696671de"
Oct 30 22:17:44.856510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:44.856470    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:49.780986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:49.780945    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:51.188536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.188503    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"]
Oct 30 22:17:51.189096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.189081    5028 cpu_manager.go:414] "containerMap: removing container" podUID="275f769a-1442-46b7-b905-b88c56205c49" containerName="revoke-keys"
Oct 30 22:17:51.189153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.189098    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275f769a-1442-46b7-b905-b88c56205c49" containerName="revoke-keys"
Oct 30 22:17:51.189153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.189147    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a3a24cae-60e8-4f64-8db0-296929f4cb92" containerName="revoke-keys"
Oct 30 22:17:51.189214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.189153    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a24cae-60e8-4f64-8db0-296929f4cb92" containerName="revoke-keys"
Oct 30 22:17:51.189263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.189254    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a24cae-60e8-4f64-8db0-296929f4cb92" containerName="revoke-keys"
Oct 30 22:17:51.189295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.189264    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="275f769a-1442-46b7-b905-b88c56205c49" containerName="revoke-keys"
Oct 30 22:17:51.194144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.194126    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.223079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.223043    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsb8c\" (UniqueName: \"kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.223276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.223207    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.223276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.223264    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.248597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.248562    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"]
Oct 30 22:17:51.289794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.289762    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:17:51.289988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.289860    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:51.289988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.289945    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:17:51.291470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.291452    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:17:51.324483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.324449    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.324637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.324495    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.324801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.324769    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsb8c\" (UniqueName: \"kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.325401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.325379    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.326898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.326880    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.384109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.384074    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsb8c\" (UniqueName: \"kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c\") pod \"maas-api-revoke-sub-76ffce53-lxmfl\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.508987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.508940    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:51.881608 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:51.881578    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c172f1_c67c_4cfc_a481_ffbba5c84c95.slice/crio-f78733e532c7cbbe01dbb1f646ef2d3e2def3c0e0487ee331b8356b01e8d8698 WatchSource:0}: Error finding container f78733e532c7cbbe01dbb1f646ef2d3e2def3c0e0487ee331b8356b01e8d8698: Status 404 returned error can't find the container with id f78733e532c7cbbe01dbb1f646ef2d3e2def3c0e0487ee331b8356b01e8d8698
Oct 30 22:17:51.882468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:51.882441    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"]
Oct 30 22:17:52.197426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:52.197388    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl" event={"ID":"30c172f1-c67c-4cfc-a481-ffbba5c84c95","Type":"ContainerStarted","Data":"afde380fdf21af20b9b2696b63f52d7234da30827d46abc0c550fca0745f0cf7"}
Oct 30 22:17:52.197916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:52.197445    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl" event={"ID":"30c172f1-c67c-4cfc-a481-ffbba5c84c95","Type":"ContainerStarted","Data":"f78733e532c7cbbe01dbb1f646ef2d3e2def3c0e0487ee331b8356b01e8d8698"}
Oct 30 22:17:52.318366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:52.318309    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl" podStartSLOduration=1.318295288 podStartE2EDuration="1.318295288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:52.317103651 +0000 UTC m=+1003.166616629" watchObservedRunningTime="2026-10-30 22:17:52.318295288 +0000 UTC m=+1003.167808239"
Oct 30 22:17:53.206847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:53.206811    5028 generic.go:356] "Generic (PLEG): container finished" podID="30c172f1-c67c-4cfc-a481-ffbba5c84c95" containerID="afde380fdf21af20b9b2696b63f52d7234da30827d46abc0c550fca0745f0cf7" exitCode=0
Oct 30 22:17:53.207378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:53.206888    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl" event={"ID":"30c172f1-c67c-4cfc-a481-ffbba5c84c95","Type":"ContainerDied","Data":"afde380fdf21af20b9b2696b63f52d7234da30827d46abc0c550fca0745f0cf7"}
Oct 30 22:17:54.374661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.374631    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:54.462074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.462042    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca\") pod \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") "
Oct 30 22:17:54.462279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.462136    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca\") pod \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") "
Oct 30 22:17:54.462279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.462232    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c\" (UniqueName: \"kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c\") pod \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\" (UID: \"30c172f1-c67c-4cfc-a481-ffbba5c84c95\") "
Oct 30 22:17:54.462568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.462525    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca" pod "30c172f1-c67c-4cfc-a481-ffbba5c84c95" (UID: "30c172f1-c67c-4cfc-a481-ffbba5c84c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:54.462747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.462730    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:54.464394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.464373    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca" pod "30c172f1-c67c-4cfc-a481-ffbba5c84c95" (UID: "30c172f1-c67c-4cfc-a481-ffbba5c84c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:54.464491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.464470    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c" pod "30c172f1-c67c-4cfc-a481-ffbba5c84c95" (UID: "30c172f1-c67c-4cfc-a481-ffbba5c84c95"). InnerVolumeSpecName "kube-api-access-rsb8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:54.564325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.564290    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsb8c\" (UniqueName: \"kubernetes.io/projected/30c172f1-c67c-4cfc-a481-ffbba5c84c95-kube-api-access-rsb8c\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:54.564325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.564322    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c172f1-c67c-4cfc-a481-ffbba5c84c95-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:54.603908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:54.603871    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:55.224031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.223993    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl" event={"ID":"30c172f1-c67c-4cfc-a481-ffbba5c84c95","Type":"ContainerDied","Data":"f78733e532c7cbbe01dbb1f646ef2d3e2def3c0e0487ee331b8356b01e8d8698"}
Oct 30 22:17:55.224031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.224013    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"
Oct 30 22:17:55.224315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.224045    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78733e532c7cbbe01dbb1f646ef2d3e2def3c0e0487ee331b8356b01e8d8698"
Oct 30 22:17:55.443377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.443339    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"]
Oct 30 22:17:55.443966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.443951    5028 cpu_manager.go:414] "containerMap: removing container" podUID="30c172f1-c67c-4cfc-a481-ffbba5c84c95" containerName="revoke-keys"
Oct 30 22:17:55.443966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.443967    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c172f1-c67c-4cfc-a481-ffbba5c84c95" containerName="revoke-keys"
Oct 30 22:17:55.444075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.444065    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c172f1-c67c-4cfc-a481-ffbba5c84c95" containerName="revoke-keys"
Oct 30 22:17:55.447728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.447707    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.475629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.475551    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7jn\" (UniqueName: \"kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.475629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.475601    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.475845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.475774    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.498353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.498319    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:17:55.498353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.498326    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:17:55.498554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.498399    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:55.498554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.498480    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:17:55.503725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.503679    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"]
Oct 30 22:17:55.577270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.577220    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7jn\" (UniqueName: \"kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.577472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.577288    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.577472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.577357    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.578021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.577996    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.579884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.579854    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.636816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.636778    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7jn\" (UniqueName: \"kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn\") pod \"maas-api-revoke-sub-036988bc-vjkqn\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:55.762098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:55.762008    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:56.142307 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:17:56.142265    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae4ee13_fe7d_4b1b_bbc0_23c625fed66d.slice/crio-2739529f1b415c10a15cece314a91d5aaf059cc25d8fce58a6c6d55437872b01 WatchSource:0}: Error finding container 2739529f1b415c10a15cece314a91d5aaf059cc25d8fce58a6c6d55437872b01: Status 404 returned error can't find the container with id 2739529f1b415c10a15cece314a91d5aaf059cc25d8fce58a6c6d55437872b01
Oct 30 22:17:56.146254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:56.146152    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"]
Oct 30 22:17:56.233409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:56.233368    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn" event={"ID":"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d","Type":"ContainerStarted","Data":"2739529f1b415c10a15cece314a91d5aaf059cc25d8fce58a6c6d55437872b01"}
Oct 30 22:17:57.242836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:57.242802    5028 generic.go:356] "Generic (PLEG): container finished" podID="dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" containerID="782c64074ebc2c4f9a5e10405a837c02c4baab0db6f47b713f862d64771b572c" exitCode=0
Oct 30 22:17:57.243268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:57.242889    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn" event={"ID":"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d","Type":"ContainerDied","Data":"782c64074ebc2c4f9a5e10405a837c02c4baab0db6f47b713f862d64771b572c"}
Oct 30 22:17:58.423670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.423643    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:58.440858 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:17:58.440820    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:17:58.510619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.510531    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca\") pod \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") "
Oct 30 22:17:58.510790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.510640    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn\" (UniqueName: \"kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn\") pod \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") "
Oct 30 22:17:58.510790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.510667    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca\") pod \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\" (UID: \"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d\") "
Oct 30 22:17:58.510916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.510889    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca" pod "dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" (UID: "dae4ee13-fe7d-4b1b-bbc0-23c625fed66d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:58.511133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.511118    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:58.512837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.512812    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca" pod "dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" (UID: "dae4ee13-fe7d-4b1b-bbc0-23c625fed66d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:58.512968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.512955    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn" pod "dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" (UID: "dae4ee13-fe7d-4b1b-bbc0-23c625fed66d"). InnerVolumeSpecName "kube-api-access-fs7jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:58.612469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.612427    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7jn\" (UniqueName: \"kubernetes.io/projected/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-kube-api-access-fs7jn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:58.612469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:58.612461    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:59.262352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:59.262318    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"
Oct 30 22:17:59.262515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:59.262349    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn" event={"ID":"dae4ee13-fe7d-4b1b-bbc0-23c625fed66d","Type":"ContainerDied","Data":"2739529f1b415c10a15cece314a91d5aaf059cc25d8fce58a6c6d55437872b01"}
Oct 30 22:17:59.262515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:59.262395    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2739529f1b415c10a15cece314a91d5aaf059cc25d8fce58a6c6d55437872b01"
Oct 30 22:17:59.382855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:17:59.382821    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.051512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.051473    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"]
Oct 30 22:18:02.052928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.052903    5028 cpu_manager.go:414] "containerMap: removing container" podUID="dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" containerName="revoke-keys"
Oct 30 22:18:02.053076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.053064    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" containerName="revoke-keys"
Oct 30 22:18:02.053312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.053301    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" containerName="revoke-keys"
Oct 30 22:18:02.059162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.059114    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.110493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.110465    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:02.110714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.110549    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:18:02.158293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.158253    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:18:02.158293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.158282    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:18:02.163004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.162977    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"]
Oct 30 22:18:02.220460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.220413    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.220698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.220524    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6bc\" (UniqueName: \"kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.220698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.220592    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.322225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.322131    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.322488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.322305    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.322488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.322363    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6bc\" (UniqueName: \"kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.323479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.323443    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.324854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.324831    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.381656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.381625    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6bc\" (UniqueName: \"kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc\") pod \"maas-api-revoke-sub-d5f57d2e-48ht5\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:02.679259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:02.679160    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:03.056858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.056817    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"]
Oct 30 22:18:03.063395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.063365    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.064166 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:18:03.064136    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276c00e4_36c5_4810_a610_fc842cdab9bd.slice/crio-8d2b5360fbabc6730ec6561144d19ec29e11a4add76b7c3b46122c0b52e1fa90 WatchSource:0}: Error finding container 8d2b5360fbabc6730ec6561144d19ec29e11a4add76b7c3b46122c0b52e1fa90: Status 404 returned error can't find the container with id 8d2b5360fbabc6730ec6561144d19ec29e11a4add76b7c3b46122c0b52e1fa90
Oct 30 22:18:03.101724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.101682    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"]
Oct 30 22:18:03.118924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.118888    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"]
Oct 30 22:18:03.232621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.232569    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.232847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.232641    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.232847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.232809    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.232971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.232849    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.232971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.232924    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g75v\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.233080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.232969    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.233080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.233046    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.233190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.233080    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.233190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.233105    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.330013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.329924    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5" event={"ID":"276c00e4-36c5-4810-a610-fc842cdab9bd","Type":"ContainerStarted","Data":"8d2b5360fbabc6730ec6561144d19ec29e11a4add76b7c3b46122c0b52e1fa90"}
Oct 30 22:18:03.334727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.334663    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.334870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.334792    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.334870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.334843    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.334978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.334881    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.334978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.334906    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.334978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.334945    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.335125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.335056    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.335125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.335101    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.335765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.335141    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g75v\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.336356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.336162    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.337425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.336860    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.337425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.337158    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.337425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.337384    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.338013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.337992    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.339021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.338953    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.339494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.339471    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.394892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.394859    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.395048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.395010    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g75v\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.574072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.574027    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"]
Oct 30 22:18:03.578404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.578377    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.628661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.628633    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-be17e7-openshift-default-dockercfg-bb9z8\""
Oct 30 22:18:03.638847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.638815    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"]
Oct 30 22:18:03.690400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.690354    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:03.739098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739048    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739281    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739378    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739410    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlw6\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739444    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739544    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739569    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739598    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.739703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.739669    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.840720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.840582    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.840720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.840672    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.840944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.840733    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.840944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.840777    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.840944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.840809    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.840944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.840883    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.841109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.841022    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.841109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.841059    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.841109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.841090    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlw6\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.841926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.841894    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.842092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.841958    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.842327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.842299    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.842408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.842390    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.842460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.842421    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.844392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.844366    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.844652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.844631    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.900006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.899973    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:03.900173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:03.900111    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlw6\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:04.127736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.127700    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"]
Oct 30 22:18:04.156836 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:18:04.156787    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc864e332_7d36_4997_a38e_e2da5014288e.slice/crio-433bb1d9ad75cdcff3ad7b93cc6f88aadf867c8ad827b87e872de0e2cb921d32 WatchSource:0}: Error finding container 433bb1d9ad75cdcff3ad7b93cc6f88aadf867c8ad827b87e872de0e2cb921d32: Status 404 returned error can't find the container with id 433bb1d9ad75cdcff3ad7b93cc6f88aadf867c8ad827b87e872de0e2cb921d32
Oct 30 22:18:04.159525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.159489    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:04.159625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.159557    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:04.159625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.159602    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:04.193518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.193490    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:04.345767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.344645    5028 generic.go:356] "Generic (PLEG): container finished" podID="276c00e4-36c5-4810-a610-fc842cdab9bd" containerID="da81eaa016ce52abad8f9e35c278c5deeb46eb0ca634b2188e6ad11772235b2f" exitCode=0
Oct 30 22:18:04.345767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.344917    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5" event={"ID":"276c00e4-36c5-4810-a610-fc842cdab9bd","Type":"ContainerDied","Data":"da81eaa016ce52abad8f9e35c278c5deeb46eb0ca634b2188e6ad11772235b2f"}
Oct 30 22:18:04.347380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.347264    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" event={"ID":"c864e332-7d36-4997-a38e-e2da5014288e","Type":"ContainerStarted","Data":"9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9"}
Oct 30 22:18:04.347380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.347320    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" event={"ID":"c864e332-7d36-4997-a38e-e2da5014288e","Type":"ContainerStarted","Data":"433bb1d9ad75cdcff3ad7b93cc6f88aadf867c8ad827b87e872de0e2cb921d32"}
Oct 30 22:18:04.405602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.405521    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.635112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.634201    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podStartSLOduration=1.634179489 podStartE2EDuration="1.634179489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:04.576746608 +0000 UTC m=+1015.426259582" watchObservedRunningTime="2026-10-30 22:18:04.634179489 +0000 UTC m=+1015.483692463"
Oct 30 22:18:04.639226 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:18:04.639179    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ac1dbd_e936_4d43_8086_1dd727ada958.slice/crio-2cfdbc611ca6160b128a191e4e59c4963e5112f94c945a7adb64cd6269078f93 WatchSource:0}: Error finding container 2cfdbc611ca6160b128a191e4e59c4963e5112f94c945a7adb64cd6269078f93: Status 404 returned error can't find the container with id 2cfdbc611ca6160b128a191e4e59c4963e5112f94c945a7adb64cd6269078f93
Oct 30 22:18:04.648191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.643598    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:04.648191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.643711    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:04.648191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.643765    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:04.648191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.647387    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"]
Oct 30 22:18:04.690477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.690449    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:04.880360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:04.880321    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:05.364320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.364282    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" event={"ID":"b2ac1dbd-e936-4d43-8086-1dd727ada958","Type":"ContainerStarted","Data":"7524e44f8975ec5517db7f8ea26de596e269c37337d018ffa586e816f8144a31"}
Oct 30 22:18:05.364806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.364350    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" event={"ID":"b2ac1dbd-e936-4d43-8086-1dd727ada958","Type":"ContainerStarted","Data":"2cfdbc611ca6160b128a191e4e59c4963e5112f94c945a7adb64cd6269078f93"}
Oct 30 22:18:05.486113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.486049    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podStartSLOduration=2.48603433 podStartE2EDuration="2.48603433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:05.483746914 +0000 UTC m=+1016.333259883" watchObservedRunningTime="2026-10-30 22:18:05.48603433 +0000 UTC m=+1016.335547354"
Oct 30 22:18:05.581608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.581582    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:05.669053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.668942    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca\") pod \"276c00e4-36c5-4810-a610-fc842cdab9bd\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") "
Oct 30 22:18:05.669204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.669063    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca\") pod \"276c00e4-36c5-4810-a610-fc842cdab9bd\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") "
Oct 30 22:18:05.669204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.669102    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc\" (UniqueName: \"kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc\") pod \"276c00e4-36c5-4810-a610-fc842cdab9bd\" (UID: \"276c00e4-36c5-4810-a610-fc842cdab9bd\") "
Oct 30 22:18:05.669405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.669372    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca" pod "276c00e4-36c5-4810-a610-fc842cdab9bd" (UID: "276c00e4-36c5-4810-a610-fc842cdab9bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:05.669701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.669667    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.677678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.677642    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc" pod "276c00e4-36c5-4810-a610-fc842cdab9bd" (UID: "276c00e4-36c5-4810-a610-fc842cdab9bd"). InnerVolumeSpecName "kube-api-access-sm6bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:05.677913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.677889    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca" pod "276c00e4-36c5-4810-a610-fc842cdab9bd" (UID: "276c00e4-36c5-4810-a610-fc842cdab9bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:05.697999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.697256    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:18:05.697999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.697319    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:18:05.782727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.773584    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276c00e4-36c5-4810-a610-fc842cdab9bd-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.782727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:05.773626    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6bc\" (UniqueName: \"kubernetes.io/projected/276c00e4-36c5-4810-a610-fc842cdab9bd-kube-api-access-sm6bc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:06.193745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.193704    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:06.195553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.195527    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:18:06.195707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.195572    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 22:18:06.387492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.387449    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"
Oct 30 22:18:06.388549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.388507    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5" event={"ID":"276c00e4-36c5-4810-a610-fc842cdab9bd","Type":"ContainerDied","Data":"8d2b5360fbabc6730ec6561144d19ec29e11a4add76b7c3b46122c0b52e1fa90"}
Oct 30 22:18:06.388672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.388578    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2b5360fbabc6730ec6561144d19ec29e11a4add76b7c3b46122c0b52e1fa90"
Oct 30 22:18:06.691716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.691658    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:18:06.691979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:06.691953    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:18:07.195087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:07.195049    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:18:07.195274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:07.195101    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 22:18:07.691416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:07.691375    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:18:07.691968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:07.691435    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:18:08.194205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:08.194164    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:18:08.194421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:08.194222    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 22:18:08.690760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:08.690726    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:18:08.690948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:08.690776    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:18:09.194240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:09.194201    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:18:09.194659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:09.194259    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 22:18:09.314467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:09.314423    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:09.690815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:09.690778    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:18:09.691005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:09.690830    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:18:10.194381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:10.194348    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:18:10.194777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:10.194399    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 22:18:10.690949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:10.690909    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:18:10.691139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:10.690995    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 22:18:11.194475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:11.194427    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 22:18:11.195028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:11.194498    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 22:18:11.695838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:11.695799    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:11.696248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:11.696226    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:11.697080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:11.697054    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:18:12.198962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:12.198930    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:12.199381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:12.199360    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:12.200027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:12.200009    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:18:13.269982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:13.269944    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:14.263972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:14.263931    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:19.080002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:19.079944    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:19.656479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:19.656434    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:23.977031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:23.976995    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:26.697785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:26.697748    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:28.436845 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:18:28.436806    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:18:28.878009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:28.877975    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:33.672190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:33.672143    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:38.132392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:38.132345    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:38.534252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:38.534210    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:43.469954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:43.469912    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:45.541984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:45.541929    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:48.287154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:48.287119    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:53.181698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:53.181649    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:54.980989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:54.980948    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:56.575566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:56.575526    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:58.238004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:18:58.237971    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:58.435501 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:18:58.435465    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:19:03.413438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:03.413402    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:05.217622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:05.217584    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:05.454065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:05.454026    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:08.583217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:08.583183    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:13.602881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:13.602836    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.292350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.292316    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"]
Oct 30 22:19:16.294712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.293668    5028 cpu_manager.go:414] "containerMap: removing container" podUID="276c00e4-36c5-4810-a610-fc842cdab9bd" containerName="revoke-keys"
Oct 30 22:19:16.294712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.293739    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276c00e4-36c5-4810-a610-fc842cdab9bd" containerName="revoke-keys"
Oct 30 22:19:16.294712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.293924    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="276c00e4-36c5-4810-a610-fc842cdab9bd" containerName="revoke-keys"
Oct 30 22:19:16.299959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.299929    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.345445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.345401    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.345754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.345727    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.345940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.345920    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.346161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.346141    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mh2b\" (UniqueName: \"kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.346292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.346276    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.346456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.346440    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.393904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.393870    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"]
Oct 30 22:19:16.412968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.412930    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.414866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.414840    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"openshift-service-ca.crt\""
Oct 30 22:19:16.415051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.414865    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"kube-root-ca.crt\""
Oct 30 22:19:16.415265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.414943    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"distinct-w3-23f3d4-kserve-self-signed-certs\""
Oct 30 22:19:16.415332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.415034    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"default-dockercfg-6kxxw\""
Oct 30 22:19:16.454723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.452156    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.454723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.453639    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.454723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.453856    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.454723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.453909    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.454723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.454065    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mh2b\" (UniqueName: \"kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.454723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.454519    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.456342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.455842    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.456660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.456639    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.467320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.463329    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.481238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.481158    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.510927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.510846    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.536922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.536860    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mh2b\" (UniqueName: \"kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b\") pod \"distinct-w3-23f3d4-kserve-5447b767c4-plp84\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:16.617356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:16.617256    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:17.020860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.020828    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"]
Oct 30 22:19:17.021192 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:19:17.021168    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b42215_85eb_46a2_9951_944feed99b07.slice/crio-f79a7a8972c05407456b3f34d6a741abaceab794828b1022097f016f3915479f WatchSource:0}: Error finding container f79a7a8972c05407456b3f34d6a741abaceab794828b1022097f016f3915479f: Status 404 returned error can't find the container with id f79a7a8972c05407456b3f34d6a741abaceab794828b1022097f016f3915479f
Oct 30 22:19:17.914713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.914657    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"]
Oct 30 22:19:17.919143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.919126    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.973199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.973168    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b13755-openshift-default-dockercfg-45x5j\""
Oct 30 22:19:17.977036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977008    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977050    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pgr\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977077    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977155    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977220    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977243    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977306    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977443    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.977497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.977480    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:17.988464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.988431    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" event={"ID":"44b42215-85eb-46a2-9951-944feed99b07","Type":"ContainerStarted","Data":"94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4"}
Oct 30 22:19:17.988605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:17.988481    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" event={"ID":"44b42215-85eb-46a2-9951-944feed99b07","Type":"ContainerStarted","Data":"f79a7a8972c05407456b3f34d6a741abaceab794828b1022097f016f3915479f"}
Oct 30 22:19:18.022862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.022822    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"]
Oct 30 22:19:18.079113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079075    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079132    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079186    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079225    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pgr\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079264    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079288    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079381    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079405    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079445    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079637    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079796    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.079901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079856    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.080009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.079984    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.080395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.080372    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.081868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.081845    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.082023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.082003    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.142650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.142611    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.143097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.143074    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pgr\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.233370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.233336    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:18.611719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.611599    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:18.617519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.617494    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:18.673888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.673591    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podStartSLOduration=2.673568977 podStartE2EDuration="2.673568977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:18.157965022 +0000 UTC m=+1089.007477995" watchObservedRunningTime="2026-10-30 22:19:18.673568977 +0000 UTC m=+1089.523081957"
Oct 30 22:19:18.680196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.680158    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"]
Oct 30 22:19:18.681371 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:19:18.681267    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc229eb45_6e20_4d24_bb47_dc47daa3dc0a.slice/crio-981211fdba9571ca8240efca719bae14022a8eb362f4779a7622127819cdd7a7 WatchSource:0}: Error finding container 981211fdba9571ca8240efca719bae14022a8eb362f4779a7622127819cdd7a7: Status 404 returned error can't find the container with id 981211fdba9571ca8240efca719bae14022a8eb362f4779a7622127819cdd7a7
Oct 30 22:19:18.683982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.683947    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:19:18.684118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.684019    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:19:18.684118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.684062    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:19:18.998638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.998593    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" event={"ID":"c229eb45-6e20-4d24-bb47-dc47daa3dc0a","Type":"ContainerStarted","Data":"7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb"}
Oct 30 22:19:18.999149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:18.998658    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" event={"ID":"c229eb45-6e20-4d24-bb47-dc47daa3dc0a","Type":"ContainerStarted","Data":"981211fdba9571ca8240efca719bae14022a8eb362f4779a7622127819cdd7a7"}
Oct 30 22:19:19.119308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:19.119243    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podStartSLOduration=2.119222835 podStartE2EDuration="2.119222835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:19.114776417 +0000 UTC m=+1089.964289383" watchObservedRunningTime="2026-10-30 22:19:19.119222835 +0000 UTC m=+1089.968735809"
Oct 30 22:19:19.237711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:19.237086    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:19.247714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:19.241826    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:19.247714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:19.241893    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:20.242234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:20.242189    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:20.242734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:20.242270    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:21.236808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:21.236771    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:21.237084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:21.237054    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:22.234709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:22.234156    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:22.234709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:22.234222    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:23.242739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:23.236106    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:23.242739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:23.236172    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:23.478725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:23.478674    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:24.238713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:24.237082    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:24.238713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:24.237146    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:25.234359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:25.234320    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:19:25.234895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:25.234377    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 22:19:26.238950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:26.238908    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:26.239384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:26.239322    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:26.240090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:26.240059    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:19:26.618325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:26.618217    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:26.645266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:26.645233    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:27.102288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:27.102251    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:19:28.374956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:28.374925    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:28.457779 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:19:28.457672    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:19:30.103778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:30.103743    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:30.263678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:30.263635    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:33.264808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:33.264770    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:33.668043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:33.667956    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:38.095144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:38.095106    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:41.165149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:41.165106    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:41.314520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:41.314487    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:43.383126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:43.383092    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:48.149389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:48.149345    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:52.749247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:52.749157    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:57.341371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:19:57.341338    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:58.436310 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:19:58.436273    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:20:01.706935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.706888    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"]
Oct 30 22:20:01.712299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.712279    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.813771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.813656    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"distinct-2-w0-d2d4c3-kserve-self-signed-certs\""
Oct 30 22:20:01.818226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.818200    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"]
Oct 30 22:20:01.847902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.847862    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.848105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.847995    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.848105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.848026    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkxx\" (UniqueName: \"kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.848193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.848118    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.848228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.848218    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.848329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.848315    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.853940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.853889    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:01.949483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949440    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.949648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949540    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.949648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949636    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.949861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949672    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkxx\" (UniqueName: \"kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.949861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949776    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.949978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949944    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.950030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.950015    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.950083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.949978    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.950673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.950647    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.952534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.952509    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:01.952653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:01.952634    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:02.014304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:02.014266    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkxx\" (UniqueName: \"kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx\") pod \"distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:02.025347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:02.025313    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:02.062390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:02.062359    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:02.410638 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:20:02.410611    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod669e0382_43ca_47fe_9926_4d9634ce133f.slice/crio-ccd2294ba7911864efdd9cf6c7633f381da79484e038d58b55bcfa03eb0f6c86 WatchSource:0}: Error finding container ccd2294ba7911864efdd9cf6c7633f381da79484e038d58b55bcfa03eb0f6c86: Status 404 returned error can't find the container with id ccd2294ba7911864efdd9cf6c7633f381da79484e038d58b55bcfa03eb0f6c86
Oct 30 22:20:02.411307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:02.411282    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"]
Oct 30 22:20:03.397174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:03.397132    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" event={"ID":"669e0382-43ca-47fe-9926-4d9634ce133f","Type":"ContainerStarted","Data":"139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754"}
Oct 30 22:20:03.397678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:03.397185    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" event={"ID":"669e0382-43ca-47fe-9926-4d9634ce133f","Type":"ContainerStarted","Data":"ccd2294ba7911864efdd9cf6c7633f381da79484e038d58b55bcfa03eb0f6c86"}
Oct 30 22:20:03.516819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:03.516763    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podStartSLOduration=2.5167480319999997 podStartE2EDuration="2.516748032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:03.516257594 +0000 UTC m=+1134.365770549" watchObservedRunningTime="2026-10-30 22:20:03.516748032 +0000 UTC m=+1134.366261005"
Oct 30 22:20:04.025873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:04.025840    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:05.558681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:05.558637    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:06.684820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:06.684767    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:11.376299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:11.376259    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:12.026212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:12.026178    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:12.042700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:12.042652    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:12.491194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:12.491163    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:20:16.071849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:16.071789    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:20.643015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:20.642976    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:21.137622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.137571    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hwbh2-294gv"]
Oct 30 22:20:21.142321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.142291    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.196809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.196775    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hwbh2-294gv"]
Oct 30 22:20:21.213561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.213523    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.213758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.213600    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.213893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.213864    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.214016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.213994    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4cmp\" (UniqueName: \"kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.315622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.315578    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.315817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.315646    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4cmp\" (UniqueName: \"kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.315817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.315724    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.315817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.315809    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.316181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.316161    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.316370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.316350    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.316442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.316424    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.375205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.375169    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4cmp\" (UniqueName: \"kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp\") pod \"periodic-gathering-hwbh2-294gv\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") " pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.459054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.459021    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:20:21.840506 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:20:21.840464    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb1f2e7_b3dd_4014_a94c_2df7e1d688a1.slice/crio-02a17ae66a505840f4e0b73cdc410dd36a926ded47bfa6e090cbdf31aac513a5 WatchSource:0}: Error finding container 02a17ae66a505840f4e0b73cdc410dd36a926ded47bfa6e090cbdf31aac513a5: Status 404 returned error can't find the container with id 02a17ae66a505840f4e0b73cdc410dd36a926ded47bfa6e090cbdf31aac513a5
Oct 30 22:20:21.840962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:21.840940    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hwbh2-294gv"]
Oct 30 22:20:22.589609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:22.589571    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwbh2-294gv" event={"ID":"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1","Type":"ContainerStarted","Data":"9764879a7a352861f28ab7abde5acb0caf2b7869a5a4a63023a1d6c99c8be9c9"}
Oct 30 22:20:22.589817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:22.589651    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwbh2-294gv" event={"ID":"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1","Type":"ContainerStarted","Data":"02a17ae66a505840f4e0b73cdc410dd36a926ded47bfa6e090cbdf31aac513a5"}
Oct 30 22:20:22.706936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:22.706879    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hwbh2-294gv" podStartSLOduration=1.706861408 podStartE2EDuration="1.706861408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:22.703153279 +0000 UTC m=+1153.552666253" watchObservedRunningTime="2026-10-30 22:20:22.706861408 +0000 UTC m=+1153.556374380"
Oct 30 22:20:25.249848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:25.249812    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:28.438453 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:20:28.438407    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:20:29.744432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:29.744392    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:34.065767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:34.065730    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"]
Oct 30 22:20:34.112660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:34.112616    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cee2dd-d772p"]
Oct 30 22:20:34.325128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:34.325041    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:35.291524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:35.291489    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a86e4bc2-746c-4f3b-aa01-653c1cbbe71d" path="/var/lib/kubelet/pods/a86e4bc2-746c-4f3b-aa01-653c1cbbe71d/volumes"
Oct 30 22:20:38.847967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:38.847926    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:43.395213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:43.395158    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:47.316125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:47.316076    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:48.006903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:48.006864    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.675998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:52.675956    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.831293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:52.831254    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:53.230224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.230188    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"]
Oct 30 22:20:53.241504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.241472    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.298591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.298559    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:53.302438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.302406    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"]
Oct 30 22:20:53.332927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.332890    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:20:53.332927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.332896    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:20:53.333202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.332894    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:20:53.373898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.373852    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4xw\" (UniqueName: \"kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.374104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.373926    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.374104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.374059    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.475576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.475537    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4xw\" (UniqueName: \"kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.475821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.475603    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.475821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.475720    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.476425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.476393    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.478247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.478223    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.534776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.534661    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4xw\" (UniqueName: \"kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw\") pod \"maas-api-revoke-sub-2b26e826-7zkb4\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.556734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.556671    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:53.934826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:53.934795    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"]
Oct 30 22:20:53.935262 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:20:53.934813    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b89f52_b35e_4364_b77a_1d966eeb6874.slice/crio-8b95b4d15ad894db5e06d85a096a5fa42bb4c808d2a04cb3906395b7b8c6a479 WatchSource:0}: Error finding container 8b95b4d15ad894db5e06d85a096a5fa42bb4c808d2a04cb3906395b7b8c6a479: Status 404 returned error can't find the container with id 8b95b4d15ad894db5e06d85a096a5fa42bb4c808d2a04cb3906395b7b8c6a479
Oct 30 22:20:54.863034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:54.863002    5028 generic.go:356] "Generic (PLEG): container finished" podID="83b89f52-b35e-4364-b77a-1d966eeb6874" containerID="4cc186c5326ca96dc9278cfcd5d82193549b6c6771852fec6f3e2c41bd6cc66e" exitCode=0
Oct 30 22:20:54.863218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:54.863093    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4" event={"ID":"83b89f52-b35e-4364-b77a-1d966eeb6874","Type":"ContainerDied","Data":"4cc186c5326ca96dc9278cfcd5d82193549b6c6771852fec6f3e2c41bd6cc66e"}
Oct 30 22:20:54.863218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:54.863182    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4" event={"ID":"83b89f52-b35e-4364-b77a-1d966eeb6874","Type":"ContainerStarted","Data":"8b95b4d15ad894db5e06d85a096a5fa42bb4c808d2a04cb3906395b7b8c6a479"}
Oct 30 22:20:56.042341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.042314    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:56.107948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.107913    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw\" (UniqueName: \"kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw\") pod \"83b89f52-b35e-4364-b77a-1d966eeb6874\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") "
Oct 30 22:20:56.108122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.108012    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca\") pod \"83b89f52-b35e-4364-b77a-1d966eeb6874\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") "
Oct 30 22:20:56.108196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.108166    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca\") pod \"83b89f52-b35e-4364-b77a-1d966eeb6874\" (UID: \"83b89f52-b35e-4364-b77a-1d966eeb6874\") "
Oct 30 22:20:56.108509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.108483    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca" pod "83b89f52-b35e-4364-b77a-1d966eeb6874" (UID: "83b89f52-b35e-4364-b77a-1d966eeb6874"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:56.108671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.108653    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:56.110185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.110161    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw" pod "83b89f52-b35e-4364-b77a-1d966eeb6874" (UID: "83b89f52-b35e-4364-b77a-1d966eeb6874"). InnerVolumeSpecName "kube-api-access-gp4xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:56.110283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.110182    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca" pod "83b89f52-b35e-4364-b77a-1d966eeb6874" (UID: "83b89f52-b35e-4364-b77a-1d966eeb6874"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:56.210297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.210267    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4xw\" (UniqueName: \"kubernetes.io/projected/83b89f52-b35e-4364-b77a-1d966eeb6874-kube-api-access-gp4xw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:56.210297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.210296    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b89f52-b35e-4364-b77a-1d966eeb6874-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:56.850655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.850614    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"]
Oct 30 22:20:56.851282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.851268    5028 cpu_manager.go:414] "containerMap: removing container" podUID="83b89f52-b35e-4364-b77a-1d966eeb6874" containerName="revoke-keys"
Oct 30 22:20:56.851329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.851283    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b89f52-b35e-4364-b77a-1d966eeb6874" containerName="revoke-keys"
Oct 30 22:20:56.851407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.851398    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b89f52-b35e-4364-b77a-1d966eeb6874" containerName="revoke-keys"
Oct 30 22:20:56.855476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.855458    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.883667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.883621    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"
Oct 30 22:20:56.883873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.883622    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4" event={"ID":"83b89f52-b35e-4364-b77a-1d966eeb6874","Type":"ContainerDied","Data":"8b95b4d15ad894db5e06d85a096a5fa42bb4c808d2a04cb3906395b7b8c6a479"}
Oct 30 22:20:56.883873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.883784    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b95b4d15ad894db5e06d85a096a5fa42bb4c808d2a04cb3906395b7b8c6a479"
Oct 30 22:20:56.910160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.910123    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"]
Oct 30 22:20:56.917300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.917266    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.917469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.917325    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.917469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.917408    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.917469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.917425    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.917469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.917455    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdvc\" (UniqueName: \"kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.917723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.917518    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:56.948875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.948835    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:56.952356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:56.952333    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"distinct-2-w6-be17e7-kserve-self-signed-certs\""
Oct 30 22:20:57.018588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.018553    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.018817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.018618    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.018817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.018729    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.019023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.018997    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.019246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.019209    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.019407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.019387    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdvc\" (UniqueName: \"kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.019585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.019565    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.019661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.019324    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.019747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.019157    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.024096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.024010    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.039922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.038032    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.078985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.078953    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdvc\" (UniqueName: \"kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc\") pod \"distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.168563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.168466    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:57.292324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.292278    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:57.555152 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:20:57.555107    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2b354fa_5137_4c50_b37e_c77ed3249c95.slice/crio-5a2eb30d967cb2d7fd37b920c4f3d5833c294ec79764004cc10a33ec950bb0e4 WatchSource:0}: Error finding container 5a2eb30d967cb2d7fd37b920c4f3d5833c294ec79764004cc10a33ec950bb0e4: Status 404 returned error can't find the container with id 5a2eb30d967cb2d7fd37b920c4f3d5833c294ec79764004cc10a33ec950bb0e4
Oct 30 22:20:57.555609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.555583    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"]
Oct 30 22:20:57.893302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.893215    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" event={"ID":"c2b354fa-5137-4c50-b37e-c77ed3249c95","Type":"ContainerStarted","Data":"10d765efc36a606ed01c60db3042b27e110e9d47e0ad30b61111ac08b4ee42d5"}
Oct 30 22:20:57.893302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:57.893256    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" event={"ID":"c2b354fa-5137-4c50-b37e-c77ed3249c95","Type":"ContainerStarted","Data":"5a2eb30d967cb2d7fd37b920c4f3d5833c294ec79764004cc10a33ec950bb0e4"}
Oct 30 22:20:58.008284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:58.008226    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podStartSLOduration=2.008211271 podStartE2EDuration="2.008211271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:58.00494942 +0000 UTC m=+1188.854462395" watchObservedRunningTime="2026-10-30 22:20:58.008211271 +0000 UTC m=+1188.857724269"
Oct 30 22:20:58.169077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:20:58.168981    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:20:58.444454 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:20:58.444354    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:21:01.861148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.861038    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:01.896604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.896558    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"]
Oct 30 22:21:01.901221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.901196    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.957449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.957417    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"]
Oct 30 22:21:01.985123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.985091    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.985282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.985158    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pvk\" (UniqueName: \"kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.985345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.985278    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.985345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.985337    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.985440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.985421    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.985479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.985450    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:01.997143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:01.997116    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"distinct-2-w4-b13755-kserve-self-signed-certs\""
Oct 30 22:21:02.031062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.031032    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:02.087658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.087620    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.087869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.087723    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pvk\" (UniqueName: \"kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.087869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.087765    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.087869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.087825    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.088051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.087888    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.088051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.087914    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.088922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.088896    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.089371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.089344    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.089812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.089790    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.092484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.092453    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.092992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.092973    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.149575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.149494    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pvk\" (UniqueName: \"kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk\") pod \"distinct-2-w4-b13755-kserve-545d9ff47f-954dg\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.217071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.217033    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:02.654524 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:02.654493    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe2ea3ec_c577_4480_b681_6c2af33669fc.slice/crio-792934417ddb1ba64dc6c4e29c5bdde056f4100f4e621308e437ba381a21a781 WatchSource:0}: Error finding container 792934417ddb1ba64dc6c4e29c5bdde056f4100f4e621308e437ba381a21a781: Status 404 returned error can't find the container with id 792934417ddb1ba64dc6c4e29c5bdde056f4100f4e621308e437ba381a21a781
Oct 30 22:21:02.654649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.654531    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"]
Oct 30 22:21:02.940172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.940049    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" event={"ID":"fe2ea3ec-c577-4480-b681-6c2af33669fc","Type":"ContainerStarted","Data":"ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93"}
Oct 30 22:21:02.940172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:02.940116    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" event={"ID":"fe2ea3ec-c577-4480-b681-6c2af33669fc","Type":"ContainerStarted","Data":"792934417ddb1ba64dc6c4e29c5bdde056f4100f4e621308e437ba381a21a781"}
Oct 30 22:21:03.062485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:03.062412    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podStartSLOduration=2.062390253 podStartE2EDuration="2.062390253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:03.058319218 +0000 UTC m=+1193.907832180" watchObservedRunningTime="2026-10-30 22:21:03.062390253 +0000 UTC m=+1193.911903228"
Oct 30 22:21:03.218483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:03.218444    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:05.050742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:05.050704    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"]
Oct 30 22:21:05.098074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:05.098038    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6832394-2b48g"]
Oct 30 22:21:05.292246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:05.292212    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e44a52-3b3e-448e-9dc8-a55417a57a56" path="/var/lib/kubelet/pods/f4e44a52-3b3e-448e-9dc8-a55417a57a56/volumes"
Oct 30 22:21:05.593742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:05.593666    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:05.900790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:05.895070    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:06.649136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:06.649090    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:07.169364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:07.169324    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:21:07.182457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:07.182424    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:21:07.997380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:07.997346    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:21:09.526653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:09.526618    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:21:09.527398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:09.527373    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:21:11.603120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:11.603083    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.217671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:12.217637    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:12.230852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:12.230825    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:13.047507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:13.047471    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:15.044439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:15.044401    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:16.303575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:16.303532    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:20.756107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:20.756016    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:21.052908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:21.052810    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"]
Oct 30 22:21:21.099414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:21.099373    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d724c6-2t8b4"]
Oct 30 22:21:21.292548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:21.292513    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21137bf-ef4a-4233-84b4-29555de5090c" path="/var/lib/kubelet/pods/a21137bf-ef4a-4233-84b4-29555de5090c/volumes"
Oct 30 22:21:22.180213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.180174    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"]
Oct 30 22:21:22.185890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.185869    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.240126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.240082    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"]
Oct 30 22:21:22.282042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.282011    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:22.282042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.282018    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:22.282319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.282237    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:22.282319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.282264    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:21:22.283892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.283872    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.283956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.283924    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.284134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.284113    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9k2j\" (UniqueName: \"kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.385469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.385434    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.385677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.385530    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.385777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.385741    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9k2j\" (UniqueName: \"kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.386619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.386589    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.388207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.388180    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.447036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.446959    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9k2j\" (UniqueName: \"kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j\") pod \"maas-api-revoke-sub-32d38b00-gbfpl\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.500457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.500425    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:22.883538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:22.883505    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c44e6b4_178b_4df3_9500_4191238d4f1f.slice/crio-06824db495f472bd1d8bf3b1a2355f38f95717571917c7055af12db77d34c5bb WatchSource:0}: Error finding container 06824db495f472bd1d8bf3b1a2355f38f95717571917c7055af12db77d34c5bb: Status 404 returned error can't find the container with id 06824db495f472bd1d8bf3b1a2355f38f95717571917c7055af12db77d34c5bb
Oct 30 22:21:22.884534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:22.884503    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"]
Oct 30 22:21:23.126476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:23.126442    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl" event={"ID":"9c44e6b4-178b-4df3-9500-4191238d4f1f","Type":"ContainerStarted","Data":"06824db495f472bd1d8bf3b1a2355f38f95717571917c7055af12db77d34c5bb"}
Oct 30 22:21:24.136223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:24.136131    5028 generic.go:356] "Generic (PLEG): container finished" podID="9c44e6b4-178b-4df3-9500-4191238d4f1f" containerID="ee49c7e2e2482477debdfe4fcdeb611d5a8e93fcefc6290e39266bccaa3d441e" exitCode=0
Oct 30 22:21:24.136635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:24.136214    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl" event={"ID":"9c44e6b4-178b-4df3-9500-4191238d4f1f","Type":"ContainerDied","Data":"ee49c7e2e2482477debdfe4fcdeb611d5a8e93fcefc6290e39266bccaa3d441e"}
Oct 30 22:21:25.282761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.282723    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:25.301659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.301633    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:25.324194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.324156    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca\") pod \"9c44e6b4-178b-4df3-9500-4191238d4f1f\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") "
Oct 30 22:21:25.324392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.324227    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca\") pod \"9c44e6b4-178b-4df3-9500-4191238d4f1f\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") "
Oct 30 22:21:25.324392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.324254    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j\" (UniqueName: \"kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j\") pod \"9c44e6b4-178b-4df3-9500-4191238d4f1f\" (UID: \"9c44e6b4-178b-4df3-9500-4191238d4f1f\") "
Oct 30 22:21:25.325234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.325181    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca" pod "9c44e6b4-178b-4df3-9500-4191238d4f1f" (UID: "9c44e6b4-178b-4df3-9500-4191238d4f1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:25.331567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.329991    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca" pod "9c44e6b4-178b-4df3-9500-4191238d4f1f" (UID: "9c44e6b4-178b-4df3-9500-4191238d4f1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:25.332980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.332920    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j" pod "9c44e6b4-178b-4df3-9500-4191238d4f1f" (UID: "9c44e6b4-178b-4df3-9500-4191238d4f1f"). InnerVolumeSpecName "kube-api-access-n9k2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:25.425991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.425910    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:25.425991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.425942    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c44e6b4-178b-4df3-9500-4191238d4f1f-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:25.425991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:25.425954    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9k2j\" (UniqueName: \"kubernetes.io/projected/9c44e6b4-178b-4df3-9500-4191238d4f1f-kube-api-access-n9k2j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:26.155780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:26.155740    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl" event={"ID":"9c44e6b4-178b-4df3-9500-4191238d4f1f","Type":"ContainerDied","Data":"06824db495f472bd1d8bf3b1a2355f38f95717571917c7055af12db77d34c5bb"}
Oct 30 22:21:26.155968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:26.155813    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06824db495f472bd1d8bf3b1a2355f38f95717571917c7055af12db77d34c5bb"
Oct 30 22:21:26.155968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:26.155754    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"
Oct 30 22:21:28.436974 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:21:28.436928    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:21:29.841897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:29.841858    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.896528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.896494    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"]
Oct 30 22:21:32.897136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.897122    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9c44e6b4-178b-4df3-9500-4191238d4f1f" containerName="revoke-keys"
Oct 30 22:21:32.897187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.897139    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c44e6b4-178b-4df3-9500-4191238d4f1f" containerName="revoke-keys"
Oct 30 22:21:32.897275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.897264    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c44e6b4-178b-4df3-9500-4191238d4f1f" containerName="revoke-keys"
Oct 30 22:21:32.900937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.900921    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:32.955224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.955184    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"]
Oct 30 22:21:32.997088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.997057    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:32.997088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.997073    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:32.997088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.997087    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:32.999086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:32.999073    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:21:33.024453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.024417    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.024661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.024640    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.024777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.024759    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wp6\" (UniqueName: \"kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.126318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.126282    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.126548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.126442    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.126548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.126494    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wp6\" (UniqueName: \"kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.127324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.127298    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.128937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.128910    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.185457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.185377    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wp6\" (UniqueName: \"kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6\") pod \"maas-api-revoke-sub-137c9bdd-njvcd\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.219920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.219885    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:33.597165 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:33.597135    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c204a1_a5a4_4788_a86d_1e8854c46e7a.slice/crio-c711d09a58b43239183e4a6bf5978a4bb575c2649a1e50068d6ec44597e00b7c WatchSource:0}: Error finding container c711d09a58b43239183e4a6bf5978a4bb575c2649a1e50068d6ec44597e00b7c: Status 404 returned error can't find the container with id c711d09a58b43239183e4a6bf5978a4bb575c2649a1e50068d6ec44597e00b7c
Oct 30 22:21:33.597730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:33.597682    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"]
Oct 30 22:21:34.234947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.234909    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd" event={"ID":"76c204a1-a5a4-4788-a86d-1e8854c46e7a","Type":"ContainerStarted","Data":"a431d2b8e2599ab673ca2dcceaa93f20018a5c74a3112975a0b218000b7a8f2e"}
Oct 30 22:21:34.235438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.234978    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd" event={"ID":"76c204a1-a5a4-4788-a86d-1e8854c46e7a","Type":"ContainerStarted","Data":"c711d09a58b43239183e4a6bf5978a4bb575c2649a1e50068d6ec44597e00b7c"}
Oct 30 22:21:34.356079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.356030    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:34.356977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.356933    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd" podStartSLOduration=2.356916828 podStartE2EDuration="2.356916828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:34.349412244 +0000 UTC m=+1225.198925218" watchObservedRunningTime="2026-10-30 22:21:34.356916828 +0000 UTC m=+1225.206429802"
Oct 30 22:21:34.632226 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:21:34.632183    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c204a1_a5a4_4788_a86d_1e8854c46e7a.slice/crio-a431d2b8e2599ab673ca2dcceaa93f20018a5c74a3112975a0b218000b7a8f2e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:21:34.796829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.796791    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"]
Oct 30 22:21:34.801178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.801152    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.831443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.831358    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"]
Oct 30 22:21:34.836974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.836802    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.876369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.876333    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"]
Oct 30 22:21:34.927562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.927525    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"]
Oct 30 22:21:34.953729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953637    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.953729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953727    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.953995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953777    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.953995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953818    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.953995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953859    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhm6\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.953995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953910    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.953995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.953950    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954006    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954036    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954106    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954167    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.954253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954226    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.954503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954263    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954315    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.954503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954359    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954440    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmsq4\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:34.954716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954502    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:34.954716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:34.954546    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.055730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055677    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.055931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055743    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhm6\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.055931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055783    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.055931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055817    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.055931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055868    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.055931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055890    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.055951    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056003    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056062    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056102    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056148    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056148    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056184    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056264    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmsq4\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056318    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056360    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056414    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056450    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056761    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.056977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056861    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.056977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.056936    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.057232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057207    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.057303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057231    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.057303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057263    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.057628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057476    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.057628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057489    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.057628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057595    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.057873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.057742    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.059631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.059526    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.059631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.059565    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.059631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.059607    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.059960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.059941    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.115526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.115445    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.115526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.115454    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.115772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.115557    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmsq4\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.115772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.115643    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhm6\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.154697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.154644    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:35.252317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.251518    5028 generic.go:356] "Generic (PLEG): container finished" podID="76c204a1-a5a4-4788-a86d-1e8854c46e7a" containerID="a431d2b8e2599ab673ca2dcceaa93f20018a5c74a3112975a0b218000b7a8f2e" exitCode=0
Oct 30 22:21:35.252317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.251785    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd" event={"ID":"76c204a1-a5a4-4788-a86d-1e8854c46e7a","Type":"ContainerDied","Data":"a431d2b8e2599ab673ca2dcceaa93f20018a5c74a3112975a0b218000b7a8f2e"}
Oct 30 22:21:35.415255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.415163    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:35.587164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.587136    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"]
Oct 30 22:21:35.588435 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:35.588407    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74927d31_8f63_428f_bfbd_a9c517a2f117.slice/crio-318773e7c7bf15f63ff06b944cd516234936d3f7fc4a71e8e87efef750c29fb6 WatchSource:0}: Error finding container 318773e7c7bf15f63ff06b944cd516234936d3f7fc4a71e8e87efef750c29fb6: Status 404 returned error can't find the container with id 318773e7c7bf15f63ff06b944cd516234936d3f7fc4a71e8e87efef750c29fb6
Oct 30 22:21:35.591069 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.591028    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:35.591163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.591104    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:35.591163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.591138    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:35.804916 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:35.804877    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe620ffb_2caf_45a0_8042_ec52b8c24227.slice/crio-e6d8af063f0abbca56ad55c6e913f36b60c470a6cf748396c3807359955696ab WatchSource:0}: Error finding container e6d8af063f0abbca56ad55c6e913f36b60c470a6cf748396c3807359955696ab: Status 404 returned error can't find the container with id e6d8af063f0abbca56ad55c6e913f36b60c470a6cf748396c3807359955696ab
Oct 30 22:21:35.807466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.807439    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"]
Oct 30 22:21:35.808553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.808516    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:35.808663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.808595    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:35.808663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:35.808637    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:36.055200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.055159    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"]
Oct 30 22:21:36.061813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.061716    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f140febb-bmmxv"]
Oct 30 22:21:36.265771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.265611    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" event={"ID":"be620ffb-2caf-45a0-8042-ec52b8c24227","Type":"ContainerStarted","Data":"dd125ed3c92c98a3a3377ac4250ae290202c52658861437ffb729febcb0854bf"}
Oct 30 22:21:36.265771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.265679    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" event={"ID":"be620ffb-2caf-45a0-8042-ec52b8c24227","Type":"ContainerStarted","Data":"e6d8af063f0abbca56ad55c6e913f36b60c470a6cf748396c3807359955696ab"}
Oct 30 22:21:36.268017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.267989    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" event={"ID":"74927d31-8f63-428f-bfbd-a9c517a2f117","Type":"ContainerStarted","Data":"4ee43571de6db7858bd3e1ae70f5d347f6fa0f12928ef25499df23ec0657a0c5"}
Oct 30 22:21:36.268174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.268044    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" event={"ID":"74927d31-8f63-428f-bfbd-a9c517a2f117","Type":"ContainerStarted","Data":"318773e7c7bf15f63ff06b944cd516234936d3f7fc4a71e8e87efef750c29fb6"}
Oct 30 22:21:36.388164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.388100    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podStartSLOduration=2.388072468 podStartE2EDuration="2.388072468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:36.381375823 +0000 UTC m=+1227.230888796" watchObservedRunningTime="2026-10-30 22:21:36.388072468 +0000 UTC m=+1227.237585445"
Oct 30 22:21:36.415673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.415638    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:36.417138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.417090    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:36.417268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.417172    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:36.508743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.508716    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:36.537625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.537556    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podStartSLOduration=2.537533035 podStartE2EDuration="2.537533035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:36.534364252 +0000 UTC m=+1227.383877226" watchObservedRunningTime="2026-10-30 22:21:36.537533035 +0000 UTC m=+1227.387046035"
Oct 30 22:21:36.683896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.683797    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca\") pod \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") "
Oct 30 22:21:36.683896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.683896    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca\") pod \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") "
Oct 30 22:21:36.684172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.683926    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6\" (UniqueName: \"kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6\") pod \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\" (UID: \"76c204a1-a5a4-4788-a86d-1e8854c46e7a\") "
Oct 30 22:21:36.684828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.684790    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca" pod "76c204a1-a5a4-4788-a86d-1e8854c46e7a" (UID: "76c204a1-a5a4-4788-a86d-1e8854c46e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:36.685400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.685377    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.687086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.687062    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6" pod "76c204a1-a5a4-4788-a86d-1e8854c46e7a" (UID: "76c204a1-a5a4-4788-a86d-1e8854c46e7a"). InnerVolumeSpecName "kube-api-access-s7wp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:36.687264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.687249    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca" pod "76c204a1-a5a4-4788-a86d-1e8854c46e7a" (UID: "76c204a1-a5a4-4788-a86d-1e8854c46e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:36.788574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.788534    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76c204a1-a5a4-4788-a86d-1e8854c46e7a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:36.788574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:36.788569    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wp6\" (UniqueName: \"kubernetes.io/projected/76c204a1-a5a4-4788-a86d-1e8854c46e7a-kube-api-access-s7wp6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:37.157124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.157057    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:37.172881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.172838    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:37.173176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.173140    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:37.288969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.288939    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"
Oct 30 22:21:37.296285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.296252    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be24d802-13f9-46de-ae2e-07becd46d6bb" path="/var/lib/kubelet/pods/be24d802-13f9-46de-ae2e-07becd46d6bb/volumes"
Oct 30 22:21:37.296895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.296871    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd" event={"ID":"76c204a1-a5a4-4788-a86d-1e8854c46e7a","Type":"ContainerDied","Data":"c711d09a58b43239183e4a6bf5978a4bb575c2649a1e50068d6ec44597e00b7c"}
Oct 30 22:21:37.297021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.296919    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c711d09a58b43239183e4a6bf5978a4bb575c2649a1e50068d6ec44597e00b7c"
Oct 30 22:21:37.416224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.416123    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:37.416224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:37.416205    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:38.155711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:38.155224    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:38.155711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:38.155296    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:38.415931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:38.415847    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:38.416333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:38.415927    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:38.976712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:38.976205    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:39.158713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:39.158163    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.158713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:39.158245    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:39.419772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:39.419208    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.419772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:39.419329    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:40.158716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.158125    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.158716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.158193    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:40.416776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.416120    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.416776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.416183    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:40.808713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.805801    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:40.971888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.971819    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:40.991866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:40.991809    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:21:41.080980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.080861    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.095916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.095851    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.112850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.112796    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.155326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.155273    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.155642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.155597    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:41.252996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.252946    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.389560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.389451    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.417293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.417256    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.417608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.417561    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:41.482012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.481945    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.812857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.812793    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:41.816723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.813476    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:21:41.975956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.973761    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:21:41.975956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.974393    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:21:41.997721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.994313    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:21:41.997721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:41.996472    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:21:42.148226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.148122    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:42.148806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.148783    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:42.178715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.163179    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.178715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.163240    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:42.286714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.258569    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:42.286714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.259138    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:21:42.322834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.322786    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:21:42.374827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.374774    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:42.407861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.403750    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:42.431722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.408524    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:21:42.537022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.536955    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:42.562736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.562256    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:21:42.605729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.599253    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.605729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:42.599329    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:43.023714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.023351    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.289950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.289850    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:21:43.297176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.297130    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:21:43.312717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.305368    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:21:43.317765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.305667    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.338712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.328575    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.338712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.328653    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:43.416720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.410720    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.503031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.502975    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": context deadline exceeded"
Oct 30 22:21:43.556880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.556755    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.624713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.606598    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.632554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.623068    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.632852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.623789    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:43.652868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.652833    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.653108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.653084    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:43.708382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.708350    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:21:43.796064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.796033    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:21:43.890000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.889912    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:21:43.949655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.949623    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:21:43.999659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:43.999626    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:21:44.081717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.078597    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:21:44.099515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.099485    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:21:44.155250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.155162    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.155529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.155499    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 22:21:44.239415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.239383    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:44.416262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.416176    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.416504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:44.416474    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 22:21:45.335173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:45.335133    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:45.337512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:45.337484    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:45.339589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:45.339558    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:21:45.649937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:45.649845    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:45.651325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:45.651258    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:45.652858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:45.652831    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:21:47.681183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.681142    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"]
Oct 30 22:21:47.682087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.682069    5028 cpu_manager.go:414] "containerMap: removing container" podUID="76c204a1-a5a4-4788-a86d-1e8854c46e7a" containerName="revoke-keys"
Oct 30 22:21:47.682150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.682089    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c204a1-a5a4-4788-a86d-1e8854c46e7a" containerName="revoke-keys"
Oct 30 22:21:47.682301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.682290    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c204a1-a5a4-4788-a86d-1e8854c46e7a" containerName="revoke-keys"
Oct 30 22:21:47.688079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.688059    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.725943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.725903    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.726160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.725985    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mp2\" (UniqueName: \"kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.726160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.726146    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.741060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.741027    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"]
Oct 30 22:21:47.780735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.780702    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:47.781123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.781101    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:21:47.785212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.785187    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:47.785341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.785298    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:47.826963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.826921    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.827153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.827043    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.827153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.827112    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mp2\" (UniqueName: \"kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.827746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.827708    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.829363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.829340    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:47.886641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:47.886599    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mp2\" (UniqueName: \"kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2\") pod \"maas-api-revoke-sub-5e9f3095-vchsx\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:48.002261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.002227    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:48.380681 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:48.380644    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf486d15_fa6a_4e26_b54a_e4eed5865fd2.slice/crio-511ec66221a42b9aa9c66c1fe94c715d24e6000c32d96a4f2dde189a8833aa02 WatchSource:0}: Error finding container 511ec66221a42b9aa9c66c1fe94c715d24e6000c32d96a4f2dde189a8833aa02: Status 404 returned error can't find the container with id 511ec66221a42b9aa9c66c1fe94c715d24e6000c32d96a4f2dde189a8833aa02
Oct 30 22:21:48.381623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.381593    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"]
Oct 30 22:21:48.685578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.685543    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx" event={"ID":"bf486d15-fa6a-4e26-b54a-e4eed5865fd2","Type":"ContainerStarted","Data":"511ec66221a42b9aa9c66c1fe94c715d24e6000c32d96a4f2dde189a8833aa02"}
Oct 30 22:21:48.842593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.842552    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:48.927767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.927666    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"]
Oct 30 22:21:48.933397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.933371    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:48.990378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:48.990331    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"]
Oct 30 22:21:49.043782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.043708    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gm5\" (UniqueName: \"kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.043782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.043789    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.044079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.043977    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.145152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.145116    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gm5\" (UniqueName: \"kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.145331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.145184    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.145331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.145260    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.146009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.145981    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.148000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.147971    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.204605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.204539    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gm5\" (UniqueName: \"kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5\") pod \"maas-api-revoke-sub-f77f7123-2mmxr\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.250089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.250047    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:49.626602 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:21:49.626569    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a9c68d_f18b_4954_b59a_eede185a5a43.slice/crio-17029588fc6646e4056c3c9a68b8041f5ad3cddfa90fe9eee5e7ed1addc1752f WatchSource:0}: Error finding container 17029588fc6646e4056c3c9a68b8041f5ad3cddfa90fe9eee5e7ed1addc1752f: Status 404 returned error can't find the container with id 17029588fc6646e4056c3c9a68b8041f5ad3cddfa90fe9eee5e7ed1addc1752f
Oct 30 22:21:49.630063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.630037    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"]
Oct 30 22:21:49.695091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.695051    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr" event={"ID":"99a9c68d-f18b-4954-b59a-eede185a5a43","Type":"ContainerStarted","Data":"17029588fc6646e4056c3c9a68b8041f5ad3cddfa90fe9eee5e7ed1addc1752f"}
Oct 30 22:21:49.696807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.696778    5028 generic.go:356] "Generic (PLEG): container finished" podID="bf486d15-fa6a-4e26-b54a-e4eed5865fd2" containerID="11776ebb66309eba4b657097981fdf5744af0f320d3b1784c8cb2cb129d792c8" exitCode=0
Oct 30 22:21:49.696937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:49.696852    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx" event={"ID":"bf486d15-fa6a-4e26-b54a-e4eed5865fd2","Type":"ContainerDied","Data":"11776ebb66309eba4b657097981fdf5744af0f320d3b1784c8cb2cb129d792c8"}
Oct 30 22:21:50.706702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.706654    5028 generic.go:356] "Generic (PLEG): container finished" podID="99a9c68d-f18b-4954-b59a-eede185a5a43" containerID="c879e29d8fab48708859f04d8e6e8eb9eeb6eea28d9682fff6e86fbcbd7bee06" exitCode=0
Oct 30 22:21:50.707119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.706743    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr" event={"ID":"99a9c68d-f18b-4954-b59a-eede185a5a43","Type":"ContainerDied","Data":"c879e29d8fab48708859f04d8e6e8eb9eeb6eea28d9682fff6e86fbcbd7bee06"}
Oct 30 22:21:50.875013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.874986    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:50.970159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.970126    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca\") pod \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") "
Oct 30 22:21:50.970342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.970244    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2\" (UniqueName: \"kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2\") pod \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") "
Oct 30 22:21:50.970384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.970343    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca\") pod \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\" (UID: \"bf486d15-fa6a-4e26-b54a-e4eed5865fd2\") "
Oct 30 22:21:50.970712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.970655    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca" pod "bf486d15-fa6a-4e26-b54a-e4eed5865fd2" (UID: "bf486d15-fa6a-4e26-b54a-e4eed5865fd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:50.971014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.970993    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:50.972265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.972240    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca" pod "bf486d15-fa6a-4e26-b54a-e4eed5865fd2" (UID: "bf486d15-fa6a-4e26-b54a-e4eed5865fd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:50.972465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:50.972436    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2" pod "bf486d15-fa6a-4e26-b54a-e4eed5865fd2" (UID: "bf486d15-fa6a-4e26-b54a-e4eed5865fd2"). InnerVolumeSpecName "kube-api-access-84mp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:51.072286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.072246    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:51.072286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.072279    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mp2\" (UniqueName: \"kubernetes.io/projected/bf486d15-fa6a-4e26-b54a-e4eed5865fd2-kube-api-access-84mp2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:51.716412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.716380    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"
Oct 30 22:21:51.716908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.716379    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx" event={"ID":"bf486d15-fa6a-4e26-b54a-e4eed5865fd2","Type":"ContainerDied","Data":"511ec66221a42b9aa9c66c1fe94c715d24e6000c32d96a4f2dde189a8833aa02"}
Oct 30 22:21:51.716908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.716482    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="511ec66221a42b9aa9c66c1fe94c715d24e6000c32d96a4f2dde189a8833aa02"
Oct 30 22:21:51.899891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.899862    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:51.983582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.983488    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5\" (UniqueName: \"kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5\") pod \"99a9c68d-f18b-4954-b59a-eede185a5a43\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") "
Oct 30 22:21:51.983582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.983536    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca\") pod \"99a9c68d-f18b-4954-b59a-eede185a5a43\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") "
Oct 30 22:21:51.983863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.983767    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca\") pod \"99a9c68d-f18b-4954-b59a-eede185a5a43\" (UID: \"99a9c68d-f18b-4954-b59a-eede185a5a43\") "
Oct 30 22:21:51.984123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.984097    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca" pod "99a9c68d-f18b-4954-b59a-eede185a5a43" (UID: "99a9c68d-f18b-4954-b59a-eede185a5a43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:51.984330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.984303    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:51.985799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.985772    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5" pod "99a9c68d-f18b-4954-b59a-eede185a5a43" (UID: "99a9c68d-f18b-4954-b59a-eede185a5a43"). InnerVolumeSpecName "kube-api-access-q7gm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:51.985974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:51.985938    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca" pod "99a9c68d-f18b-4954-b59a-eede185a5a43" (UID: "99a9c68d-f18b-4954-b59a-eede185a5a43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:52.086152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:52.086116    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gm5\" (UniqueName: \"kubernetes.io/projected/99a9c68d-f18b-4954-b59a-eede185a5a43-kube-api-access-q7gm5\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:52.086152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:52.086146    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a9c68d-f18b-4954-b59a-eede185a5a43-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:52.726046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:52.726009    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr" event={"ID":"99a9c68d-f18b-4954-b59a-eede185a5a43","Type":"ContainerDied","Data":"17029588fc6646e4056c3c9a68b8041f5ad3cddfa90fe9eee5e7ed1addc1752f"}
Oct 30 22:21:52.726046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:52.726042    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"
Oct 30 22:21:52.726599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:52.726062    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17029588fc6646e4056c3c9a68b8041f5ad3cddfa90fe9eee5e7ed1addc1752f"
Oct 30 22:21:53.052210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.052063    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"]
Oct 30 22:21:53.052877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.052856    5028 cpu_manager.go:414] "containerMap: removing container" podUID="99a9c68d-f18b-4954-b59a-eede185a5a43" containerName="revoke-keys"
Oct 30 22:21:53.052962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.052878    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a9c68d-f18b-4954-b59a-eede185a5a43" containerName="revoke-keys"
Oct 30 22:21:53.052962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.052912    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bf486d15-fa6a-4e26-b54a-e4eed5865fd2" containerName="revoke-keys"
Oct 30 22:21:53.052962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.052917    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf486d15-fa6a-4e26-b54a-e4eed5865fd2" containerName="revoke-keys"
Oct 30 22:21:53.053108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.053020    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a9c68d-f18b-4954-b59a-eede185a5a43" containerName="revoke-keys"
Oct 30 22:21:53.053108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.053035    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf486d15-fa6a-4e26-b54a-e4eed5865fd2" containerName="revoke-keys"
Oct 30 22:21:53.056816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.056795    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.108040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.107990    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:53.108237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.108060    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:53.108796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.108772    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:53.108796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.108788    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:21:53.160234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.160200    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"]
Oct 30 22:21:53.200073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.200030    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.200311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.200287    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwt4z\" (UniqueName: \"kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.200445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.200428    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.301751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.301705    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.301958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.301793    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.301958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.301905    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwt4z\" (UniqueName: \"kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.302497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.302444    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.304199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.304174    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.342906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.342869    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:53.362726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.362666    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwt4z\" (UniqueName: \"kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z\") pod \"maas-api-revoke-sub-f27beb53-hfg2g\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.370365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.370339    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:53.784214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:53.784183    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"]
Oct 30 22:21:54.746078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:54.746041    5028 generic.go:356] "Generic (PLEG): container finished" podID="07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" containerID="6ca527349d8106d384f4db65fe0f6343de4f7d79ac6651a949cdf18a38fcd9f8" exitCode=0
Oct 30 22:21:54.746215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:54.746124    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g" event={"ID":"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a","Type":"ContainerDied","Data":"6ca527349d8106d384f4db65fe0f6343de4f7d79ac6651a949cdf18a38fcd9f8"}
Oct 30 22:21:54.746215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:54.746187    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g" event={"ID":"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a","Type":"ContainerStarted","Data":"4bec60656a3beef1760d4349c4d595507cc25d4d791f3a6b161312c58934908e"}
Oct 30 22:21:55.918953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:55.918916    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:56.040676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.040581    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca\") pod \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") "
Oct 30 22:21:56.040894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.040767    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca\") pod \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") "
Oct 30 22:21:56.040894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.040847    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z\" (UniqueName: \"kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z\") pod \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\" (UID: \"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a\") "
Oct 30 22:21:56.041409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.041371    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca" pod "07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" (UID: "07f1f6ce-d8c5-48e1-a3de-99f714f2e45a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:56.043424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.043386    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca" pod "07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" (UID: "07f1f6ce-d8c5-48e1-a3de-99f714f2e45a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:56.043572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.043552    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z" pod "07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" (UID: "07f1f6ce-d8c5-48e1-a3de-99f714f2e45a"). InnerVolumeSpecName "kube-api-access-zwt4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:56.143275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.143230    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:56.143275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.143274    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:56.143492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.143292    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwt4z\" (UniqueName: \"kubernetes.io/projected/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a-kube-api-access-zwt4z\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:56.765270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.765238    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"
Oct 30 22:21:56.765444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.765237    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g" event={"ID":"07f1f6ce-d8c5-48e1-a3de-99f714f2e45a","Type":"ContainerDied","Data":"4bec60656a3beef1760d4349c4d595507cc25d4d791f3a6b161312c58934908e"}
Oct 30 22:21:56.765444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:56.765364    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bec60656a3beef1760d4349c4d595507cc25d4d791f3a6b161312c58934908e"
Oct 30 22:21:57.053751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:57.053625    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"]
Oct 30 22:21:57.100906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:57.100869    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda3e8b-rbwh4"]
Oct 30 22:21:57.292227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:57.292195    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2978736e-5cdb-4b19-ba7a-ad66b2cb6088" path="/var/lib/kubelet/pods/2978736e-5cdb-4b19-ba7a-ad66b2cb6088/volumes"
Oct 30 22:21:57.958673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:57.958615    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:58.438364 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:21:58.438323    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:21:58.783279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:58.783245    5028 generic.go:356] "Generic (PLEG): container finished" podID="cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" containerID="9764879a7a352861f28ab7abde5acb0caf2b7869a5a4a63023a1d6c99c8be9c9" exitCode=0
Oct 30 22:21:58.783460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:58.783318    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwbh2-294gv" event={"ID":"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1","Type":"ContainerDied","Data":"9764879a7a352861f28ab7abde5acb0caf2b7869a5a4a63023a1d6c99c8be9c9"}
Oct 30 22:21:59.967036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:21:59.966992    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:22:00.096746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.096642    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle\") pod \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") "
Oct 30 22:22:00.096961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.096941    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp\" (UniqueName: \"kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp\") pod \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") "
Oct 30 22:22:00.097060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.097043    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle\") pod \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") "
Oct 30 22:22:00.097124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.097109    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path\") pod \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\" (UID: \"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1\") "
Oct 30 22:22:00.097239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.097209    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle" pod "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" (UID: "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:00.097682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.097649    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle" pod "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" (UID: "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:00.097682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.097659    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path" pod "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" (UID: "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:00.097822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.097740    5028 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-trusted-ca-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.099963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.099927    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp" pod "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" (UID: "cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1"). InnerVolumeSpecName "kube-api-access-f4cmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:00.198985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.198952    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4cmp\" (UniqueName: \"kubernetes.io/projected/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-kube-api-access-f4cmp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.198985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.198981    5028 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-service-ca-bundle\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.198985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.198991    5028 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1-archives-path\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:00.800392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.800361    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hwbh2-294gv"
Oct 30 22:22:00.800576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.800408    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hwbh2-294gv" event={"ID":"cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1","Type":"ContainerDied","Data":"02a17ae66a505840f4e0b73cdc410dd36a926ded47bfa6e090cbdf31aac513a5"}
Oct 30 22:22:00.800576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:00.800458    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02a17ae66a505840f4e0b73cdc410dd36a926ded47bfa6e090cbdf31aac513a5"
Oct 30 22:22:01.205424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.205338    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"]
Oct 30 22:22:01.205977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.205963    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" containerName="insights-gathering"
Oct 30 22:22:01.206018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.205978    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" containerName="insights-gathering"
Oct 30 22:22:01.206018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.206016    5028 cpu_manager.go:414] "containerMap: removing container" podUID="07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" containerName="revoke-keys"
Oct 30 22:22:01.206083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.206022    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" containerName="revoke-keys"
Oct 30 22:22:01.206132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.206123    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" containerName="revoke-keys"
Oct 30 22:22:01.206166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.206138    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb1f2e7-b3dd-4014-a94c-2df7e1d688a1" containerName="insights-gathering"
Oct 30 22:22:01.209968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.209948    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.264766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.264734    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"]
Oct 30 22:22:01.306576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.306545    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:01.306576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.306570    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:01.307460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.307442    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:01.307515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.307459    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:22:01.314997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.314961    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvth\" (UniqueName: \"kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.315125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.315063    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.315218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.315203    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.416207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.416170    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvth\" (UniqueName: \"kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.416414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.416227    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.416414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.416398    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.417151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.417126    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.419052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.419023    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.477261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.477231    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvth\" (UniqueName: \"kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth\") pod \"maas-api-revoke-sub-3b4513b1-7dzr4\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.525967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.525930    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:01.904310 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:01.904275    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc337aa3_a1e7_4b2c_a87e_368a02c09c66.slice/crio-24343e61006ab04e54ec203e36f8b7d9f2a0ddb0dd4f6663036b1c3e374f8661 WatchSource:0}: Error finding container 24343e61006ab04e54ec203e36f8b7d9f2a0ddb0dd4f6663036b1c3e374f8661: Status 404 returned error can't find the container with id 24343e61006ab04e54ec203e36f8b7d9f2a0ddb0dd4f6663036b1c3e374f8661
Oct 30 22:22:01.905496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.905473    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"]
Oct 30 22:22:01.916445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:01.916419    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:02.084800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:02.084755    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:02.570843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:02.570782    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:02.819752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:02.819708    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4" event={"ID":"dc337aa3-a1e7-4b2c-a87e-368a02c09c66","Type":"ContainerStarted","Data":"179c0fa162f31a840447fb8c6c4f85a3d5e1543771777a5e03943f0dff9a4327"}
Oct 30 22:22:02.819965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:02.819786    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4" event={"ID":"dc337aa3-a1e7-4b2c-a87e-368a02c09c66","Type":"ContainerStarted","Data":"24343e61006ab04e54ec203e36f8b7d9f2a0ddb0dd4f6663036b1c3e374f8661"}
Oct 30 22:22:02.935270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:02.935207    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4" podStartSLOduration=1.9351724799999999 podStartE2EDuration="1.93517248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:02.931680857 +0000 UTC m=+1253.781193830" watchObservedRunningTime="2026-10-30 22:22:02.93517248 +0000 UTC m=+1253.784685454"
Oct 30 22:22:03.350235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.350196    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"]
Oct 30 22:22:03.354591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.354568    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.416050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.416010    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"]
Oct 30 22:22:03.444019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.443984    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnxf\" (UniqueName: \"kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.444208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.444049    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.444364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.444336    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.545915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.545823    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.546061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.545988    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnxf\" (UniqueName: \"kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.546121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.546085    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.546660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.546629    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.548720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.548675    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.605339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.605307    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnxf\" (UniqueName: \"kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf\") pod \"maas-api-revoke-sub-4f53c404-wqn8d\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.668622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.668585    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:03.848102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.848002    5028 generic.go:356] "Generic (PLEG): container finished" podID="dc337aa3-a1e7-4b2c-a87e-368a02c09c66" containerID="179c0fa162f31a840447fb8c6c4f85a3d5e1543771777a5e03943f0dff9a4327" exitCode=0
Oct 30 22:22:03.848102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:03.848080    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4" event={"ID":"dc337aa3-a1e7-4b2c-a87e-368a02c09c66","Type":"ContainerDied","Data":"179c0fa162f31a840447fb8c6c4f85a3d5e1543771777a5e03943f0dff9a4327"}
Oct 30 22:22:04.011523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.011489    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"]
Oct 30 22:22:04.017491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.017464    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.027214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.027192    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"]
Oct 30 22:22:04.031752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.031724    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.057390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057352    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.057578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057482    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.057578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057518    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.057578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057550    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.057783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057586    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.057783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057617    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.057783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057668    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.057783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057708    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.057783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057725    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.057783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057756    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddcw\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057787    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057806    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057828    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057854    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057894    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057910    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057930    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.058085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.057946    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgt9\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.074785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.074750    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"]
Oct 30 22:22:04.084415 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:04.084381    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5b4300_b174_4466_8a8c_9ce221bb6a89.slice/crio-afeb899e96c302f3e52047d3718127772452254ef91826250c398436970efa68 WatchSource:0}: Error finding container afeb899e96c302f3e52047d3718127772452254ef91826250c398436970efa68: Status 404 returned error can't find the container with id afeb899e96c302f3e52047d3718127772452254ef91826250c398436970efa68
Oct 30 22:22:04.090575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.090545    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"]
Oct 30 22:22:04.129797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.129727    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"]
Oct 30 22:22:04.159851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.159810    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.159864    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.159891    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.159925    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddcw\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.159962    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.159986    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160018    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160055    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160113    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160137    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160168    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160191    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgt9\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160262    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160270    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160382    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160427    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.160815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160471    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160523    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.160815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.160557    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.161041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.161020    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.161246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.161221    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.161534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.161509    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.161795    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.162210    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.162256    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.162330    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.162637    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.163147    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.165634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.164535    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"]
Oct 30 22:22:04.169856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.169826    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.170030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.170008    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.170396 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.170368    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.170522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.170500    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.173215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.173194    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.241999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.241950    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"]
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.263984    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264049    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264138    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264207    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264223    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbg4\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264249    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264276    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264317    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.264559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.264387    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.325154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.325085    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.325830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.325806    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.326991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.326945    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddcw\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.334204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.333928    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:04.339484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.336303    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgt9\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.367052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.354961    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367599    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367738    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367823    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367850    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbg4\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367883    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367922    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.367974    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.368066    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.368163    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.368599    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.369223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.368883    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.399656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.377897    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.399656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.378191    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.399656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.393899    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.446019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.445977    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.452948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.452410    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.456880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.456848    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbg4\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.460268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.460237    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.500426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.499279    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:04.936176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:04.910815    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"]
Oct 30 22:22:05.018738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.005609    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.081709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.081142    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"]
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134179    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134226    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134266    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134293    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134319    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134360    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfs5r\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134457    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134489    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.138306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.134568    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.198276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.198185    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:05.247221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.247167    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:05.247698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.247639    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:05.249057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.249029    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:05.312479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.312446    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"]
Oct 30 22:22:05.411707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.395149    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.464001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.463965    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.464592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.449415    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.567377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.494714    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:05.578807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.503319    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.742719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.737819    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:05.742719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.737923    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:05.742719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.579066    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.742719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.738489    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.742719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.738533    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrkt\" (UniqueName: \"kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:05.742719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.738589    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfs5r\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.752433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.751853    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.752433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.751924    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:05.752433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.751985    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.752433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.752148    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.752433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.752222    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:05.772719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.759155    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.791898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.791863    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.820359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.820324    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.927451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.927416    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.928052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.928024    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.929064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.929043    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:05.968271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.949442    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrkt\" (UniqueName: \"kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:05.968271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.949604    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:05.985547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:05.985457    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:06.041713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.033938    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfs5r\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:06.135717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.099080    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:06.135717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.099931    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:06.135717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.128913    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d" event={"ID":"6e5b4300-b174-4466-8a8c-9ce221bb6a89","Type":"ContainerStarted","Data":"1654d3d79b4cd5362646e44df0f591da4ea412c185cb76afffe3b5caae917938"}
Oct 30 22:22:06.135717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.128979    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d" event={"ID":"6e5b4300-b174-4466-8a8c-9ce221bb6a89","Type":"ContainerStarted","Data":"afeb899e96c302f3e52047d3718127772452254ef91826250c398436970efa68"}
Oct 30 22:22:06.135717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.129012    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"]
Oct 30 22:22:06.135717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.129045    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"]
Oct 30 22:22:06.234549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.234505    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrkt\" (UniqueName: \"kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:06.325764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.323176    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4338d8-7x5kw\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:06.616792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.610551    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:06.644494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.644433    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:06.683714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.655788    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:06.740304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.685734    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 30 22:22:06.869770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.853432    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:06.869770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.853550    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:06.869770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.853598    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:06.913713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.900969    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.913888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.913769    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:06.999827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:06.999765    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.038970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.034576    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"]
Oct 30 22:22:07.043453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.043412    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.264719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.105934    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" event={"ID":"b40e1899-0698-432a-9c93-784212968f3f","Type":"ContainerStarted","Data":"6abca6e9a3bf848f2bccee8da2c4b9a613c5839d853c2c3c23e6d275ea881205"}
Oct 30 22:22:07.267868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.267835    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"]
Oct 30 22:22:07.268082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.268056    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" event={"ID":"a1503661-b46b-4acb-98eb-a97d754419e1","Type":"ContainerStarted","Data":"7741dc255e569477d8f59a23160e1bf801a7a386c70bd0484badf720e3eb04ed"}
Oct 30 22:22:07.268246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.228964    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:07.362171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.362133    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca\") pod \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") "
Oct 30 22:22:07.362884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.362858    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth\" (UniqueName: \"kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth\") pod \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") "
Oct 30 22:22:07.363454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.363434    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca\") pod \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\" (UID: \"dc337aa3-a1e7-4b2c-a87e-368a02c09c66\") "
Oct 30 22:22:07.364745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.362752    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca" pod "dc337aa3-a1e7-4b2c-a87e-368a02c09c66" (UID: "dc337aa3-a1e7-4b2c-a87e-368a02c09c66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:07.385901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.385862    5028 generic.go:356] "Generic (PLEG): container finished" podID="6e5b4300-b174-4466-8a8c-9ce221bb6a89" containerID="1654d3d79b4cd5362646e44df0f591da4ea412c185cb76afffe3b5caae917938" exitCode=0
Oct 30 22:22:07.402782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.402639    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d" podStartSLOduration=4.402620304 podStartE2EDuration="4.402620304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:06.046263526 +0000 UTC m=+1256.895776502" watchObservedRunningTime="2026-10-30 22:22:07.402620304 +0000 UTC m=+1258.252133278"
Oct 30 22:22:07.431832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.431796    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" event={"ID":"2474dd98-1521-4f25-b10d-675ac3157105","Type":"ContainerStarted","Data":"d7e1ed949633a6a7edd827ad9bba6cb39c2c330eefd65987e28abe7c95616714"}
Oct 30 22:22:07.432067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.432049    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d" event={"ID":"6e5b4300-b174-4466-8a8c-9ce221bb6a89","Type":"ContainerDied","Data":"1654d3d79b4cd5362646e44df0f591da4ea412c185cb76afffe3b5caae917938"}
Oct 30 22:22:07.459954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.459902    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.463935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.463894    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:22:07.497416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.497378    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:07.766715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.765783    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"]
Oct 30 22:22:07.933048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.928334    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:07.933048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:07.929850    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:22:08.141387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.033904    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.208758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.121223    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth" pod "dc337aa3-a1e7-4b2c-a87e-368a02c09c66" (UID: "dc337aa3-a1e7-4b2c-a87e-368a02c09c66"). InnerVolumeSpecName "kube-api-access-5cvth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:08.208949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.142582    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca" pod "dc337aa3-a1e7-4b2c-a87e-368a02c09c66" (UID: "dc337aa3-a1e7-4b2c-a87e-368a02c09c66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:08.209074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.193779    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded"
Oct 30 22:22:08.209204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.160636    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:08.210218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.208789    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:22:08.213168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.213144    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40689d5e-gr4vm"]
Oct 30 22:22:08.214737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.214717    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:22:08.223867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.223839    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvth\" (UniqueName: \"kubernetes.io/projected/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-kube-api-access-5cvth\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:08.223995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.223872    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc337aa3-a1e7-4b2c-a87e-368a02c09c66-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:08.235021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.234992    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:22:08.316927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.316890    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.495750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.493985    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"
Oct 30 22:22:08.497571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.497396    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4" event={"ID":"dc337aa3-a1e7-4b2c-a87e-368a02c09c66","Type":"ContainerDied","Data":"24343e61006ab04e54ec203e36f8b7d9f2a0ddb0dd4f6663036b1c3e374f8661"}
Oct 30 22:22:08.497571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.497513    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24343e61006ab04e54ec203e36f8b7d9f2a0ddb0dd4f6663036b1c3e374f8661"
Oct 30 22:22:08.507425 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:08.507348    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5277e7ea_ef19_4d91_bdf7_2603d56f7a70.slice/crio-bf5537473e90f17e2520a750f6e4532a662f005883e459cfc069926def17fc1e WatchSource:0}: Error finding container bf5537473e90f17e2520a750f6e4532a662f005883e459cfc069926def17fc1e: Status 404 returned error can't find the container with id bf5537473e90f17e2520a750f6e4532a662f005883e459cfc069926def17fc1e
Oct 30 22:22:08.508913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.508884    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" event={"ID":"2474dd98-1521-4f25-b10d-675ac3157105","Type":"ContainerStarted","Data":"e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b"}
Oct 30 22:22:08.511025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.511004    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"]
Oct 30 22:22:08.514207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.514176    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" event={"ID":"b40e1899-0698-432a-9c93-784212968f3f","Type":"ContainerStarted","Data":"862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47"}
Oct 30 22:22:08.516252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.516227    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" event={"ID":"a1503661-b46b-4acb-98eb-a97d754419e1","Type":"ContainerStarted","Data":"9e80627354d0378691fb1a78ea368bd75a85621f6c9feec30762bf4c0406b98e"}
Oct 30 22:22:08.519304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.519215    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"]
Oct 30 22:22:08.519761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.519730    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:08.519851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.519795    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:08.519851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.519827    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:08.651727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.651641    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podStartSLOduration=5.651618409 podStartE2EDuration="5.651618409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:08.644894582 +0000 UTC m=+1259.494407556" watchObservedRunningTime="2026-10-30 22:22:08.651618409 +0000 UTC m=+1259.501131382"
Oct 30 22:22:08.783578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.782676    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"]
Oct 30 22:22:08.783578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.783499    5028 cpu_manager.go:414] "containerMap: removing container" podUID="dc337aa3-a1e7-4b2c-a87e-368a02c09c66" containerName="revoke-keys"
Oct 30 22:22:08.783578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.783513    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc337aa3-a1e7-4b2c-a87e-368a02c09c66" containerName="revoke-keys"
Oct 30 22:22:08.783862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.783629    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc337aa3-a1e7-4b2c-a87e-368a02c09c66" containerName="revoke-keys"
Oct 30 22:22:08.789525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.789499    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:08.790700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.790596    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:08.815189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.815149    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"]
Oct 30 22:22:08.844178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.844102    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podStartSLOduration=4.844064631 podStartE2EDuration="4.844064631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:08.837857037 +0000 UTC m=+1259.687370002" watchObservedRunningTime="2026-10-30 22:22:08.844064631 +0000 UTC m=+1259.693577605"
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.984859    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podStartSLOduration=5.984837971 podStartE2EDuration="5.984837971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:08.978407437 +0000 UTC m=+1259.827920431" watchObservedRunningTime="2026-10-30 22:22:08.984837971 +0000 UTC m=+1259.834350945"
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.987156    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca\") pod \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") "
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.987349    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca\") pod \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") "
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.987404    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf\" (UniqueName: \"kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf\") pod \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\" (UID: \"6e5b4300-b174-4466-8a8c-9ce221bb6a89\") "
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.987659    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.987874    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v46b\" (UniqueName: \"kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.987943    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.988484    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca" pod "6e5b4300-b174-4466-8a8c-9ce221bb6a89" (UID: "6e5b4300-b174-4466-8a8c-9ce221bb6a89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:08.994764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.992934    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca" pod "6e5b4300-b174-4466-8a8c-9ce221bb6a89" (UID: "6e5b4300-b174-4466-8a8c-9ce221bb6a89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:09.000493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:08.997261    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf" pod "6e5b4300-b174-4466-8a8c-9ce221bb6a89" (UID: "6e5b4300-b174-4466-8a8c-9ce221bb6a89"). InnerVolumeSpecName "kube-api-access-4cnxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.094130    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.094324    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v46b\" (UniqueName: \"kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.094383    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.094458    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.094478    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnxf\" (UniqueName: \"kubernetes.io/projected/6e5b4300-b174-4466-8a8c-9ce221bb6a89-kube-api-access-4cnxf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.094494    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5b4300-b174-4466-8a8c-9ce221bb6a89-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.095749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.095317    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.108776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.106500    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.440826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.368991    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:09.440826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.369057    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:09.440826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.389384    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:09.440826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.389445    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:09.519311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.518327    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:09.519311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.518406    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:09.632143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.632106    5028 scope.go:122] "RemoveContainer" containerID="bee05b1c1e3c6379d1ad3a13b7a3e3322d1af7a63efb7316ffce96d37f2dfab6"
Oct 30 22:22:09.694075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.693386    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v46b\" (UniqueName: \"kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b\") pod \"maas-api-revoke-sub-a9036ba6-9grsk\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.730971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.730925    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:09.770241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.770203    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819f57b9-b878-4146-b356-5d341d7625d5" path="/var/lib/kubelet/pods/819f57b9-b878-4146-b356-5d341d7625d5/volumes"
Oct 30 22:22:09.771662 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.771638    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:09.771861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.771846    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:09.772034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.772021    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:09.792770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.783579    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d" event={"ID":"6e5b4300-b174-4466-8a8c-9ce221bb6a89","Type":"ContainerDied","Data":"afeb899e96c302f3e52047d3718127772452254ef91826250c398436970efa68"}
Oct 30 22:22:09.792770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.783644    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afeb899e96c302f3e52047d3718127772452254ef91826250c398436970efa68"
Oct 30 22:22:09.792770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.783739    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"
Oct 30 22:22:09.812319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.810929    5028 scope.go:122] "RemoveContainer" containerID="5c1e4780a805dfcd8d065d2924802fb801574c3351912312882fcbf214b0a2c3"
Oct 30 22:22:09.815833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.815780    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" event={"ID":"5277e7ea-ef19-4d91-bdf7-2603d56f7a70","Type":"ContainerStarted","Data":"755e196a5498dd8ceaff6e58ea50e72febb305af7e5a9a58cfa300f9311c28e6"}
Oct 30 22:22:09.816038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.816019    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" event={"ID":"5277e7ea-ef19-4d91-bdf7-2603d56f7a70","Type":"ContainerStarted","Data":"bf5537473e90f17e2520a750f6e4532a662f005883e459cfc069926def17fc1e"}
Oct 30 22:22:09.862656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:09.862611    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw" event={"ID":"68b9de13-8797-42f8-89c0-e01761292eb5","Type":"ContainerStarted","Data":"c65a1b9425d1dba3237ab187824611499803948bc0373c5c928b7084b6d8010d"}
Oct 30 22:22:10.110727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.095382    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podStartSLOduration=6.095363005 podStartE2EDuration="6.095363005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:10.068791936 +0000 UTC m=+1260.918304910" watchObservedRunningTime="2026-10-30 22:22:10.095363005 +0000 UTC m=+1260.944875983"
Oct 30 22:22:10.201738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.196460    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:10.201738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.197369    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"]
Oct 30 22:22:10.267729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.239816    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:10.335924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.324377    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73384deb-7p6ss"]
Oct 30 22:22:10.370642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.352108    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:10.370642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.352188    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:10.395007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.377797    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:10.395007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.377850    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:10.540319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.540274    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:10.598712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.552594    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw" podStartSLOduration=6.552573919 podStartE2EDuration="6.552573919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:10.25963367 +0000 UTC m=+1261.109146644" watchObservedRunningTime="2026-10-30 22:22:10.552573919 +0000 UTC m=+1261.402086896"
Oct 30 22:22:10.704015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.571832    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:10.704015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.702976    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:10.712004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.711973    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:10.712378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.712356    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:10.714797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.714766    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:10.947772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:10.934157    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.114739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.097169    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.114739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.099072    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.159727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.158777    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 30 22:22:11.279792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.271516    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.279792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.278467    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.304728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.283424    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:22:11.304728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.303397    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:22:11.304728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.303486    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:22:11.397765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.368841    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:11.397765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.368936    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:11.413714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.412876    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:11.413714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.412953    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:11.433772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.430756    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.674768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.674251    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:11.674768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.674335    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:11.745337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.739858    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551a4fe9-7f02-49a7-86c4-e46f7d30b534" path="/var/lib/kubelet/pods/551a4fe9-7f02-49a7-86c4-e46f7d30b534/volumes"
Oct 30 22:22:11.745337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.740456    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"]
Oct 30 22:22:11.784232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.772282    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6e5b4300-b174-4466-8a8c-9ce221bb6a89" containerName="revoke-keys"
Oct 30 22:22:11.784232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.772316    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5b4300-b174-4466-8a8c-9ce221bb6a89" containerName="revoke-keys"
Oct 30 22:22:11.784232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.772761    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5b4300-b174-4466-8a8c-9ce221bb6a89" containerName="revoke-keys"
Oct 30 22:22:11.876005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.860750    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:11.939812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.860828    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:11.957807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.957756    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:11.986998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.986130    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:22:11.987204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:11.987086    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.043097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.022568    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw" event={"ID":"68b9de13-8797-42f8-89c0-e01761292eb5","Type":"ContainerStarted","Data":"8ca62d47b1a6e1551662a05b260ca667add9203c9fed9467f24356648babcaa7"}
Oct 30 22:22:12.166725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.125819    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:22:12.166725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.144513    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:22:12.169187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.169161    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:22:12.169649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.169632    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:22:12.324839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.324792    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:22:12.327537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.327507    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:22:12.329817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.329786    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:12.340709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.330238    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:22:12.361714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.346401    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:12.362279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.362237    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:12.364954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.364927    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:12.365115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.365093    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:12.391778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.391741    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"]
Oct 30 22:22:12.417059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.416751    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:22:12.459752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.449805    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.459752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.450185    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.475124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.473805    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:12.475124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.474227    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:22:12.486656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.486483    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56972\" (UniqueName: \"kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.555332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.555297    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:12.555580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.555550    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:12.604393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.604276    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.613037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.612856    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.687711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.643637    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"]
Oct 30 22:22:12.687711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.666620    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.687711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.666786    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56972\" (UniqueName: \"kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.732790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.732750    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:12.821986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.821946    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:12.967774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.895921    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56972\" (UniqueName: \"kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.967774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.938394    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f698129e-svz2j\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:12.990735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:12.976759    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:13.065711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.064856    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:13.175712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.150797    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:13.280780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.243912    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:22:13.280780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.243960    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:13.353724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.341111    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.381807    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.343197    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.343220    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.342761    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.343238    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.398196    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:13.413730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.398249    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:13.466714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.447074    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:13.489500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.488714    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:13.546898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.546807    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:13.569818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.563238    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:13.600719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.574957    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.650704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.650646    5028 generic.go:356] "Generic (PLEG): container finished" podID="68b9de13-8797-42f8-89c0-e01761292eb5" containerID="8ca62d47b1a6e1551662a05b260ca667add9203c9fed9467f24356648babcaa7" exitCode=0
Oct 30 22:22:13.734795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.728013    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:13.734795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:13.728073    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:14.087713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.059586    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:14.122729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.115261    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:22:14.122729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.115321    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw" event={"ID":"68b9de13-8797-42f8-89c0-e01761292eb5","Type":"ContainerDied","Data":"8ca62d47b1a6e1551662a05b260ca667add9203c9fed9467f24356648babcaa7"}
Oct 30 22:22:14.122729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.115397    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk" event={"ID":"363c2e60-14b3-461d-9c00-75d2af311c33","Type":"ContainerStarted","Data":"e140b395c40f4ef727da6db33b349983c66492c680a76f393e8cb30f9c48bb1b"}
Oct 30 22:22:14.291957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.279226    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:14.291957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.279768    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:22:14.330786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.329286    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"]
Oct 30 22:22:14.417710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.399448    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:14.430785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.418397    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:14.430785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.419793    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:14.430785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.418475    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:14.430785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.419856    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:14.430785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.419908    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:22:14.430785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.420121    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:22:14.651754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.651639    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:22:14.730714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.728313    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:14.730977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.730943    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:14.806958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.769642    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:14.806958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:14.784783    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:15.071711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.071094    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:22:15.150722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.141703    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:15.233894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.233786    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:15.804784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.801092    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:15.804784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.801799    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:22:15.804784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.802286    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:22:15.808624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.808575    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:15.809443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.809326    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:22:15.890756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.889770    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:15.995729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.995673    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:15.995993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.995963    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:15.997428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.997403    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:15.997700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:15.997645    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:16.119809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.119241    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.119809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.119315    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:16.346714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.305892    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.349236    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.437179    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.380039    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.437249    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.399386    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.437288    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:16.461788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.382345    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:22:16.520712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.518776    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.520712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.518841    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 22:22:16.699111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.699074    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.713276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.713177    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:16.807933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.807889    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:16.920036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.919809    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:22:16.967772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.950943    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:16.968773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.968737    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:22:16.995780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.970003    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 30 22:22:16.995780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:16.968669    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:17.012000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.001866    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:17.012000 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:22:17.009859    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363c2e60_14b3_461d_9c00_75d2af311c33.slice/crio-ed47b0e7095be4e5f54783d9026effe5b90ccdb0e4ac78a74a9f045838876957.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363c2e60_14b3_461d_9c00_75d2af311c33.slice/crio-conmon-ed47b0e7095be4e5f54783d9026effe5b90ccdb0e4ac78a74a9f045838876957.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:22:17.096078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.089345    5028 generic.go:356] "Generic (PLEG): container finished" podID="363c2e60-14b3-461d-9c00-75d2af311c33" containerID="ed47b0e7095be4e5f54783d9026effe5b90ccdb0e4ac78a74a9f045838876957" exitCode=0
Oct 30 22:22:17.097313 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:22:17.097283    5028 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.805s"
Oct 30 22:22:17.097713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.097680    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk" event={"ID":"363c2e60-14b3-461d-9c00-75d2af311c33","Type":"ContainerDied","Data":"ed47b0e7095be4e5f54783d9026effe5b90ccdb0e4ac78a74a9f045838876957"}
Oct 30 22:22:17.097861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.097848    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c852b97-qwzdj"]
Oct 30 22:22:17.097984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.097972    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j" event={"ID":"bf6b5cc7-0bf3-4143-a6d6-029ce057c870","Type":"ContainerStarted","Data":"4588ae75f22869af1ce5cca6e232d26fd3610148e74d48b6304514413fc4dd7f"}
Oct 30 22:22:17.098189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.098164    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:22:17.098363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.098341    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:22:17.099724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.099679    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:22:17.101461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.101411    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw" event={"ID":"68b9de13-8797-42f8-89c0-e01761292eb5","Type":"ContainerDied","Data":"c65a1b9425d1dba3237ab187824611499803948bc0373c5c928b7084b6d8010d"}
Oct 30 22:22:17.101638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.101608    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c65a1b9425d1dba3237ab187824611499803948bc0373c5c928b7084b6d8010d"
Oct 30 22:22:17.111599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.099386    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"
Oct 30 22:22:17.183712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.175237    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"]
Oct 30 22:22:17.183712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.176044    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:22:17.217713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.201752    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:22:17.217713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.206999    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:22:17.233853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.227012    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:22:17.290966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.274651    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca\") pod \"68b9de13-8797-42f8-89c0-e01761292eb5\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") "
Oct 30 22:22:17.290966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.274823    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt\" (UniqueName: \"kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt\") pod \"68b9de13-8797-42f8-89c0-e01761292eb5\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") "
Oct 30 22:22:17.290966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.274939    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\") pod \"68b9de13-8797-42f8-89c0-e01761292eb5\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") "
Oct 30 22:22:17.290966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.290177    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca" pod "68b9de13-8797-42f8-89c0-e01761292eb5" (UID: "68b9de13-8797-42f8-89c0-e01761292eb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:17.408079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.405345    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:17.408079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.406135    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:22:17.408079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.406184    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 22:22:17.411017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.409817    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:17.411017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.409872    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:17.492675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.485874    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e" path="/var/lib/kubelet/pods/ad2f278c-b7a2-43c1-8d8b-0ecb8d2d670e/volumes"
Oct 30 22:22:17.494553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.494520    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:22:17.515650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.515621    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:22:17.601127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.601077    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt" pod "68b9de13-8797-42f8-89c0-e01761292eb5" (UID: "68b9de13-8797-42f8-89c0-e01761292eb5"). InnerVolumeSpecName "kube-api-access-fwrkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:17.649713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.643048    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca" pod "68b9de13-8797-42f8-89c0-e01761292eb5" (UID: "68b9de13-8797-42f8-89c0-e01761292eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:17.649713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.643360    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\") pod \"68b9de13-8797-42f8-89c0-e01761292eb5\" (UID: \"68b9de13-8797-42f8-89c0-e01761292eb5\") "
Oct 30 22:22:17.649713 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:17.645113    5028 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/68b9de13-8797-42f8-89c0-e01761292eb5/volumes/kubernetes.io~secret/maas-api-xks-ca
Oct 30 22:22:17.649713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.645131    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca" pod "68b9de13-8797-42f8-89c0-e01761292eb5" (UID: "68b9de13-8797-42f8-89c0-e01761292eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:17.655380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.654132    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b9de13-8797-42f8-89c0-e01761292eb5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:17.655380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.654169    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrkt\" (UniqueName: \"kubernetes.io/projected/68b9de13-8797-42f8-89c0-e01761292eb5-kube-api-access-fwrkt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:17.691038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.691001    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:17.691313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:17.691281    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:18.118846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.117848    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j" event={"ID":"bf6b5cc7-0bf3-4143-a6d6-029ce057c870","Type":"ContainerStarted","Data":"a63883d78825c346c23a794ebb16c4ffcf42aafdd1042500ec507404ec9e81b6"}
Oct 30 22:22:18.155080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.154983    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:18.245097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.245051    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"]
Oct 30 22:22:18.250484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.250458    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78a71015-n7qmd"]
Oct 30 22:22:18.382721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.379959    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 22:22:18.394837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.394782    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 22:22:18.495717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.494170    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:18.507482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.504308    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:18.609745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.609416    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:22:18.698708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.697476    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:22:18.698708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.697532    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 22:22:18.909859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.907238    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:18.909859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.909400    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:18.912899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:18.912144    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:22:19.162287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.161053    5028 generic.go:356] "Generic (PLEG): container finished" podID="bf6b5cc7-0bf3-4143-a6d6-029ce057c870" containerID="a63883d78825c346c23a794ebb16c4ffcf42aafdd1042500ec507404ec9e81b6" exitCode=0
Oct 30 22:22:19.164733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.162219    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j" event={"ID":"bf6b5cc7-0bf3-4143-a6d6-029ce057c870","Type":"ContainerDied","Data":"a63883d78825c346c23a794ebb16c4ffcf42aafdd1042500ec507404ec9e81b6"}
Oct 30 22:22:19.168271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.167433    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j" podStartSLOduration=9.167415067 podStartE2EDuration="9.167415067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:19.008049179 +0000 UTC m=+1269.857562152" watchObservedRunningTime="2026-10-30 22:22:19.167415067 +0000 UTC m=+1270.016928045"
Oct 30 22:22:19.252858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.239863    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.328259    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c117f7e-8536-4c56-b21c-5f166ed2383a" path="/var/lib/kubelet/pods/6c117f7e-8536-4c56-b21c-5f166ed2383a/volumes"
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.350048    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca\") pod \"363c2e60-14b3-461d-9c00-75d2af311c33\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") "
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.350219    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b\" (UniqueName: \"kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b\") pod \"363c2e60-14b3-461d-9c00-75d2af311c33\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") "
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.350263    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca\") pod \"363c2e60-14b3-461d-9c00-75d2af311c33\" (UID: \"363c2e60-14b3-461d-9c00-75d2af311c33\") "
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.351762    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca" pod "363c2e60-14b3-461d-9c00-75d2af311c33" (UID: "363c2e60-14b3-461d-9c00-75d2af311c33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.363962    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b" pod "363c2e60-14b3-461d-9c00-75d2af311c33" (UID: "363c2e60-14b3-461d-9c00-75d2af311c33"). InnerVolumeSpecName "kube-api-access-9v46b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:19.378140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.366233    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca" pod "363c2e60-14b3-461d-9c00-75d2af311c33" (UID: "363c2e60-14b3-461d-9c00-75d2af311c33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:19.406880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.406828    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:19.415233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.413532    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:19.421342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.421271    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:22:19.457298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.457260    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:19.457517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.457504    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v46b\" (UniqueName: \"kubernetes.io/projected/363c2e60-14b3-461d-9c00-75d2af311c33-kube-api-access-9v46b\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:19.457598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.457588    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c2e60-14b3-461d-9c00-75d2af311c33-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:19.575944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.575618    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"]
Oct 30 22:22:19.576990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.576959    5028 cpu_manager.go:414] "containerMap: removing container" podUID="68b9de13-8797-42f8-89c0-e01761292eb5" containerName="revoke-keys"
Oct 30 22:22:19.576990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.576986    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b9de13-8797-42f8-89c0-e01761292eb5" containerName="revoke-keys"
Oct 30 22:22:19.577191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.577005    5028 cpu_manager.go:414] "containerMap: removing container" podUID="363c2e60-14b3-461d-9c00-75d2af311c33" containerName="revoke-keys"
Oct 30 22:22:19.577191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.577012    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363c2e60-14b3-461d-9c00-75d2af311c33" containerName="revoke-keys"
Oct 30 22:22:19.577302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.577238    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="363c2e60-14b3-461d-9c00-75d2af311c33" containerName="revoke-keys"
Oct 30 22:22:19.577302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.577269    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b9de13-8797-42f8-89c0-e01761292eb5" containerName="revoke-keys"
Oct 30 22:22:19.582177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.582153    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.639617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.639582    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"]
Oct 30 22:22:19.664573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.664467    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.664781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.664650    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.664956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.664927    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.665040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.665022    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.665189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.665170    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.665261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.665243    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtl9l\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.665418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.665400    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.665474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.665443    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.665547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.665530    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.761383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.761342    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"]
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767238    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767352    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767382    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767449    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767476    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtl9l\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767553    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767581    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767612    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.767785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.767726    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.768380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.768260    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.769271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.768522    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.769271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.768837    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.769271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.769045    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.769271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.769105    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.769271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.769238    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.771937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.771912    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.772201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.772186    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.830312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.818621    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"]
Oct 30 22:22:19.830312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.829806    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.840779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.840748    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtl9l\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.870527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.868745    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.870527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.869070    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.870527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.869104    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2c6v\" (UniqueName: \"kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.900058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.900028    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:19.903285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.903176    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:19.904918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.904565    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:19.906157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.905730    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:22:19.971081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.971044    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.971200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.971083    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2c6v\" (UniqueName: \"kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.971200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.971175    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.972829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.972800    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:19.977351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:19.976912    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:20.030453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.030416    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2c6v\" (UniqueName: \"kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v\") pod \"maas-api-revoke-sub-300f715d-xvkss\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:20.085394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.085311    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:20.185874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.185843    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"
Oct 30 22:22:20.189534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.189502    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk" event={"ID":"363c2e60-14b3-461d-9c00-75d2af311c33","Type":"ContainerDied","Data":"e140b395c40f4ef727da6db33b349983c66492c680a76f393e8cb30f9c48bb1b"}
Oct 30 22:22:20.189669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.189556    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e140b395c40f4ef727da6db33b349983c66492c680a76f393e8cb30f9c48bb1b"
Oct 30 22:22:20.346834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.345554    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"]
Oct 30 22:22:20.354146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.353539    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:20.354146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.353620    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:20.354146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.353661    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:20.452811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.452784    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:20.493858 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:20.493817    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ae36f2_05bc_4e3f_b23e_351b4dcc069b.slice/crio-97bea8981c956893b891d8b69fc71e2f96b4933d2ce6908a5db3681d7c2a455b WatchSource:0}: Error finding container 97bea8981c956893b891d8b69fc71e2f96b4933d2ce6908a5db3681d7c2a455b: Status 404 returned error can't find the container with id 97bea8981c956893b891d8b69fc71e2f96b4933d2ce6908a5db3681d7c2a455b
Oct 30 22:22:20.495356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.495330    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"]
Oct 30 22:22:20.594258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.594166    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca\") pod \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") "
Oct 30 22:22:20.594258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.594228    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca\") pod \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") "
Oct 30 22:22:20.594514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.594304    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972\" (UniqueName: \"kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972\") pod \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\" (UID: \"bf6b5cc7-0bf3-4143-a6d6-029ce057c870\") "
Oct 30 22:22:20.594634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.594601    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca" pod "bf6b5cc7-0bf3-4143-a6d6-029ce057c870" (UID: "bf6b5cc7-0bf3-4143-a6d6-029ce057c870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:20.595037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.595016    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:20.596574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.596550    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972" pod "bf6b5cc7-0bf3-4143-a6d6-029ce057c870" (UID: "bf6b5cc7-0bf3-4143-a6d6-029ce057c870"). InnerVolumeSpecName "kube-api-access-56972". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:20.596819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.596794    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca" pod "bf6b5cc7-0bf3-4143-a6d6-029ce057c870" (UID: "bf6b5cc7-0bf3-4143-a6d6-029ce057c870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:20.635940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.635905    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"]
Oct 30 22:22:20.636612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.636597    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bf6b5cc7-0bf3-4143-a6d6-029ce057c870" containerName="revoke-keys"
Oct 30 22:22:20.636612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.636612    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6b5cc7-0bf3-4143-a6d6-029ce057c870" containerName="revoke-keys"
Oct 30 22:22:20.636793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.636757    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6b5cc7-0bf3-4143-a6d6-029ce057c870" containerName="revoke-keys"
Oct 30 22:22:20.642102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.642082    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.696196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.696162    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"]
Oct 30 22:22:20.696478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.696455    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.696567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.696485    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjpf\" (UniqueName: \"kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.696736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.696718    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.696823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.696806    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:20.696823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.696819    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56972\" (UniqueName: \"kubernetes.io/projected/bf6b5cc7-0bf3-4143-a6d6-029ce057c870-kube-api-access-56972\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:20.798253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.798203    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.799562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.798492    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.799562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.798528    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjpf\" (UniqueName: \"kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.800532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.800504    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.801759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.801736    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.857663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.857579    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjpf\" (UniqueName: \"kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf\") pod \"maas-api-revoke-sub-b86ed724-c9bqk\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:20.957933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:20.957898    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:21.064807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.064777    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"]
Oct 30 22:22:21.074030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.073592    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1672ea5e-gtznt"]
Oct 30 22:22:21.192720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.192653    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" event={"ID":"800b1992-b5d7-4eda-a700-b6c3807ecf0a","Type":"ContainerStarted","Data":"57766c24ff4414f368653a887c96a85f306f8319a9b7c086ad360910e02b232a"}
Oct 30 22:22:21.192720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.192724    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" event={"ID":"800b1992-b5d7-4eda-a700-b6c3807ecf0a","Type":"ContainerStarted","Data":"40614e6ba0af55f851fd619c6c1cec286ea71abbfcadf4579a900dc547528e4b"}
Oct 30 22:22:21.194418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.194391    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"
Oct 30 22:22:21.194594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.194431    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j" event={"ID":"bf6b5cc7-0bf3-4143-a6d6-029ce057c870","Type":"ContainerDied","Data":"4588ae75f22869af1ce5cca6e232d26fd3610148e74d48b6304514413fc4dd7f"}
Oct 30 22:22:21.194594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.194490    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4588ae75f22869af1ce5cca6e232d26fd3610148e74d48b6304514413fc4dd7f"
Oct 30 22:22:21.196202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.196181    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss" event={"ID":"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b","Type":"ContainerStarted","Data":"a54fb226827d3396fe0b0a89b9fc0f13ee2560d7e393344ce354aca31dfd01d8"}
Oct 30 22:22:21.196300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.196217    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss" event={"ID":"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b","Type":"ContainerStarted","Data":"97bea8981c956893b891d8b69fc71e2f96b4933d2ce6908a5db3681d7c2a455b"}
Oct 30 22:22:21.294221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.294185    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720ff68f-55a1-44b7-9b5f-028180c2ff4e" path="/var/lib/kubelet/pods/720ff68f-55a1-44b7-9b5f-028180c2ff4e/volumes"
Oct 30 22:22:21.329122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.324894    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podStartSLOduration=2.324874588 podStartE2EDuration="2.324874588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:21.315994652 +0000 UTC m=+1272.165507619" watchObservedRunningTime="2026-10-30 22:22:21.324874588 +0000 UTC m=+1272.174387563"
Oct 30 22:22:21.379390 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:21.379361    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9947f535_998c_4fb2_b6e9_f98fae674a36.slice/crio-339aa461c2932d04f5f8ef68d1d7fd97cd89c801d3f43897d618913fd8a00bd2 WatchSource:0}: Error finding container 339aa461c2932d04f5f8ef68d1d7fd97cd89c801d3f43897d618913fd8a00bd2: Status 404 returned error can't find the container with id 339aa461c2932d04f5f8ef68d1d7fd97cd89c801d3f43897d618913fd8a00bd2
Oct 30 22:22:21.380895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.380864    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"]
Oct 30 22:22:21.500413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.500340    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss" podStartSLOduration=2.500319396 podStartE2EDuration="2.500319396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:21.496812356 +0000 UTC m=+1272.346325329" watchObservedRunningTime="2026-10-30 22:22:21.500319396 +0000 UTC m=+1272.349832371"
Oct 30 22:22:21.900700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.900647    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:21.902292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.902266    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:22:21.902413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:21.902311    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:22:22.207391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.207291    5028 generic.go:356] "Generic (PLEG): container finished" podID="f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" containerID="a54fb226827d3396fe0b0a89b9fc0f13ee2560d7e393344ce354aca31dfd01d8" exitCode=0
Oct 30 22:22:22.207391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.207332    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss" event={"ID":"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b","Type":"ContainerDied","Data":"a54fb226827d3396fe0b0a89b9fc0f13ee2560d7e393344ce354aca31dfd01d8"}
Oct 30 22:22:22.209449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.209424    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk" event={"ID":"9947f535-998c-4fb2-b6e9-f98fae674a36","Type":"ContainerStarted","Data":"363a9f3552bdbfe4137004e312b54d1c54ef03347402ac57ce7b4fba8adb1824"}
Oct 30 22:22:22.209585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.209476    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk" event={"ID":"9947f535-998c-4fb2-b6e9-f98fae674a36","Type":"ContainerStarted","Data":"339aa461c2932d04f5f8ef68d1d7fd97cd89c801d3f43897d618913fd8a00bd2"}
Oct 30 22:22:22.470412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.470360    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk" podStartSLOduration=2.470346449 podStartE2EDuration="2.470346449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:22.465041482 +0000 UTC m=+1273.314554454" watchObservedRunningTime="2026-10-30 22:22:22.470346449 +0000 UTC m=+1273.319859422"
Oct 30 22:22:22.901147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.901063    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:22:22.901147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.901116    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:22:22.910074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:22.910047    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:23.097658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.097623    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"]
Oct 30 22:22:23.105578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.105545    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5299232c-5js65"]
Oct 30 22:22:23.253034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.252970    5028 generic.go:356] "Generic (PLEG): container finished" podID="9947f535-998c-4fb2-b6e9-f98fae674a36" containerID="363a9f3552bdbfe4137004e312b54d1c54ef03347402ac57ce7b4fba8adb1824" exitCode=0
Oct 30 22:22:23.254707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.254651    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk" event={"ID":"9947f535-998c-4fb2-b6e9-f98fae674a36","Type":"ContainerDied","Data":"363a9f3552bdbfe4137004e312b54d1c54ef03347402ac57ce7b4fba8adb1824"}
Oct 30 22:22:23.296580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.296109    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fb77e5-0f21-4a94-a12b-fea8fbbc7558" path="/var/lib/kubelet/pods/58fb77e5-0f21-4a94-a12b-fea8fbbc7558/volumes"
Oct 30 22:22:23.499421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.499395    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:23.671625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.671534    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca\") pod \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") "
Oct 30 22:22:23.671625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.671574    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v\" (UniqueName: \"kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v\") pod \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") "
Oct 30 22:22:23.671625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.671603    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca\") pod \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\" (UID: \"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b\") "
Oct 30 22:22:23.672039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.672015    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca" pod "f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" (UID: "f7ae36f2-05bc-4e3f-b23e-351b4dcc069b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:23.672378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.672355    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:23.674470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.674440    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca" pod "f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" (UID: "f7ae36f2-05bc-4e3f-b23e-351b4dcc069b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:23.675000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.674968    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v" pod "f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" (UID: "f7ae36f2-05bc-4e3f-b23e-351b4dcc069b"). InnerVolumeSpecName "kube-api-access-z2c6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:23.778709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.776986    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:23.778709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.777030    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2c6v\" (UniqueName: \"kubernetes.io/projected/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b-kube-api-access-z2c6v\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:23.900813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.900779    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:22:23.900974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:23.900842    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:22:24.251797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.251759    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"]
Oct 30 22:22:24.252454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.252438    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" containerName="revoke-keys"
Oct 30 22:22:24.252512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.252455    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" containerName="revoke-keys"
Oct 30 22:22:24.252607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.252597    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" containerName="revoke-keys"
Oct 30 22:22:24.256607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.256587    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.264210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.264180    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"
Oct 30 22:22:24.264330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.264226    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss" event={"ID":"f7ae36f2-05bc-4e3f-b23e-351b4dcc069b","Type":"ContainerDied","Data":"97bea8981c956893b891d8b69fc71e2f96b4933d2ce6908a5db3681d7c2a455b"}
Oct 30 22:22:24.264330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.264270    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97bea8981c956893b891d8b69fc71e2f96b4933d2ce6908a5db3681d7c2a455b"
Oct 30 22:22:24.308376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.308329    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"]
Oct 30 22:22:24.385086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.385052    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.385277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.385102    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wnq\" (UniqueName: \"kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.385277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.385229    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.486669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.486636    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wnq\" (UniqueName: \"kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.486851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.486708    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.487066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.487042    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.487905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.487886    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.489593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.489572    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.547749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.547718    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wnq\" (UniqueName: \"kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq\") pod \"maas-api-revoke-sub-e98951e5-rgx7l\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.549543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.549516    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"]
Oct 30 22:22:24.553463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.553437    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:24.554298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.554282    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.573007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.572979    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:24.604414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.604181    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"]
Oct 30 22:22:24.693650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.693264    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca\") pod \"9947f535-998c-4fb2-b6e9-f98fae674a36\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") "
Oct 30 22:22:24.693650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.693396    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca\") pod \"9947f535-998c-4fb2-b6e9-f98fae674a36\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") "
Oct 30 22:22:24.693856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.693715    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf\" (UniqueName: \"kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf\") pod \"9947f535-998c-4fb2-b6e9-f98fae674a36\" (UID: \"9947f535-998c-4fb2-b6e9-f98fae674a36\") "
Oct 30 22:22:24.694546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.694388    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.694546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.694473    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbqc\" (UniqueName: \"kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.697216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.695007    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.697216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.696553    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca" pod "9947f535-998c-4fb2-b6e9-f98fae674a36" (UID: "9947f535-998c-4fb2-b6e9-f98fae674a36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:24.699427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.699148    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf" pod "9947f535-998c-4fb2-b6e9-f98fae674a36" (UID: "9947f535-998c-4fb2-b6e9-f98fae674a36"). InnerVolumeSpecName "kube-api-access-rhjpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:24.700636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.699808    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca" pod "9947f535-998c-4fb2-b6e9-f98fae674a36" (UID: "9947f535-998c-4fb2-b6e9-f98fae674a36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:24.796185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.796077    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.796185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.796159    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbqc\" (UniqueName: \"kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.796516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.796493    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.796602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.796590    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:24.796657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.796607    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9947f535-998c-4fb2-b6e9-f98fae674a36-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:24.796657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.796621    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhjpf\" (UniqueName: \"kubernetes.io/projected/9947f535-998c-4fb2-b6e9-f98fae674a36-kube-api-access-rhjpf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:24.798122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.797329    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.799965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.799916    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.857007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.856978    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbqc\" (UniqueName: \"kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc\") pod \"maas-api-revoke-sub-d47f0567-zwrnk\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.875587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.875558    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:24.900643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.900605    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:22:24.900788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:24.900667    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:22:25.014659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.014627    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"]
Oct 30 22:22:25.275923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.275897    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"
Oct 30 22:22:25.276521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.275907    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk" event={"ID":"9947f535-998c-4fb2-b6e9-f98fae674a36","Type":"ContainerDied","Data":"339aa461c2932d04f5f8ef68d1d7fd97cd89c801d3f43897d618913fd8a00bd2"}
Oct 30 22:22:25.276615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.276558    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339aa461c2932d04f5f8ef68d1d7fd97cd89c801d3f43897d618913fd8a00bd2"
Oct 30 22:22:25.281706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.281654    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l" event={"ID":"84c0158a-8d75-4f4c-bfbf-7fea099c1192","Type":"ContainerStarted","Data":"fcafd09cad449721a0ad7d3a6e8b50f849d0bd0c37d188a3224cce7d004ff0fb"}
Oct 30 22:22:25.293956 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:25.293894    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261995b5_8ff5_48ce_b0d2_a3c4f3bf0e47.slice/crio-240eb66b544e8fe0e0a103ca40f7d8d397b0f1529ecd2eb2eabdb729aa7f5ddc WatchSource:0}: Error finding container 240eb66b544e8fe0e0a103ca40f7d8d397b0f1529ecd2eb2eabdb729aa7f5ddc: Status 404 returned error can't find the container with id 240eb66b544e8fe0e0a103ca40f7d8d397b0f1529ecd2eb2eabdb729aa7f5ddc
Oct 30 22:22:25.302345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.302320    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"]
Oct 30 22:22:25.901412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.901375    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:22:25.901587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:25.901433    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:22:26.296303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.296269    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk" event={"ID":"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47","Type":"ContainerStarted","Data":"a0fee25067bfced559734b511bf0857b1f5acdc183cef3362e35bdfd2cc93bdb"}
Oct 30 22:22:26.297030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.296335    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk" event={"ID":"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47","Type":"ContainerStarted","Data":"240eb66b544e8fe0e0a103ca40f7d8d397b0f1529ecd2eb2eabdb729aa7f5ddc"}
Oct 30 22:22:26.298353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.298329    5028 generic.go:356] "Generic (PLEG): container finished" podID="84c0158a-8d75-4f4c-bfbf-7fea099c1192" containerID="640d7825605ddacc7a1cf37080f42d3f27721a14d923af37259e4eb20d4d93c8" exitCode=0
Oct 30 22:22:26.298461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.298368    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l" event={"ID":"84c0158a-8d75-4f4c-bfbf-7fea099c1192","Type":"ContainerDied","Data":"640d7825605ddacc7a1cf37080f42d3f27721a14d923af37259e4eb20d4d93c8"}
Oct 30 22:22:26.413678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.413618    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk" podStartSLOduration=2.413602416 podStartE2EDuration="2.413602416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:26.410143707 +0000 UTC m=+1277.259656680" watchObservedRunningTime="2026-10-30 22:22:26.413602416 +0000 UTC m=+1277.263115388"
Oct 30 22:22:26.901272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.901182    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:22:26.901272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:26.901255    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:22:27.050282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.050244    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"]
Oct 30 22:22:27.055268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.055237    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b253be-t678c"]
Oct 30 22:22:27.196126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.196038    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"]
Oct 30 22:22:27.196289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.196252    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy" containerID="cri-o://a8aafca25e0a7ef81b53e6793456244aa30c49ea07b7fcc2a300997fc7da1ed0" gracePeriod=30
Oct 30 22:22:27.296170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.296135    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0fa524-2bbf-4798-a6d2-14bbf43eded1" path="/var/lib/kubelet/pods/2d0fa524-2bbf-4798-a6d2-14bbf43eded1/volumes"
Oct 30 22:22:27.309897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.309853    5028 generic.go:356] "Generic (PLEG): container finished" podID="261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" containerID="a0fee25067bfced559734b511bf0857b1f5acdc183cef3362e35bdfd2cc93bdb" exitCode=0
Oct 30 22:22:27.310295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.309932    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk" event={"ID":"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47","Type":"ContainerDied","Data":"a0fee25067bfced559734b511bf0857b1f5acdc183cef3362e35bdfd2cc93bdb"}
Oct 30 22:22:27.534585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.534557    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:27.649356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.649311    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq\" (UniqueName: \"kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq\") pod \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") "
Oct 30 22:22:27.649537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.649428    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca\") pod \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") "
Oct 30 22:22:27.649537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.649458    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca\") pod \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\" (UID: \"84c0158a-8d75-4f4c-bfbf-7fea099c1192\") "
Oct 30 22:22:27.652522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.652482    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca" pod "84c0158a-8d75-4f4c-bfbf-7fea099c1192" (UID: "84c0158a-8d75-4f4c-bfbf-7fea099c1192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:27.653000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.652974    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca" pod "84c0158a-8d75-4f4c-bfbf-7fea099c1192" (UID: "84c0158a-8d75-4f4c-bfbf-7fea099c1192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:27.655607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.655582    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq" pod "84c0158a-8d75-4f4c-bfbf-7fea099c1192" (UID: "84c0158a-8d75-4f4c-bfbf-7fea099c1192"). InnerVolumeSpecName "kube-api-access-22wnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:27.694298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.694254    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:27.752566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.752470    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wnq\" (UniqueName: \"kubernetes.io/projected/84c0158a-8d75-4f4c-bfbf-7fea099c1192-kube-api-access-22wnq\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:27.752566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.752517    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:27.752566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.752537    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c0158a-8d75-4f4c-bfbf-7fea099c1192-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:27.906016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.905982    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:27.906563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.906498    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:27.907247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:27.907230    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:22:28.074709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.074614    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"]
Oct 30 22:22:28.079786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.079752    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314ad3f9-kf6ml"]
Oct 30 22:22:28.320979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.320947    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l" event={"ID":"84c0158a-8d75-4f4c-bfbf-7fea099c1192","Type":"ContainerDied","Data":"fcafd09cad449721a0ad7d3a6e8b50f849d0bd0c37d188a3224cce7d004ff0fb"}
Oct 30 22:22:28.321362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.321003    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcafd09cad449721a0ad7d3a6e8b50f849d0bd0c37d188a3224cce7d004ff0fb"
Oct 30 22:22:28.321362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.320961    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"
Oct 30 22:22:28.440377 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:22:28.440338    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:22:28.482700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.482668    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:28.563766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.563735    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca\") pod \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") "
Oct 30 22:22:28.563948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.563826    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc\" (UniqueName: \"kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc\") pod \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") "
Oct 30 22:22:28.564018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.563992    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca\") pod \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\" (UID: \"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47\") "
Oct 30 22:22:28.564194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.564171    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca" pod "261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" (UID: "261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:28.564565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.564543    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:28.566165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.566141    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc" pod "261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" (UID: "261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47"). InnerVolumeSpecName "kube-api-access-8pbqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:28.566255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.566207    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca" pod "261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" (UID: "261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:28.666158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.666069    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pbqc\" (UniqueName: \"kubernetes.io/projected/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-kube-api-access-8pbqc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:28.666158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:28.666103    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:29.294763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:29.294730    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e407743a-c859-4dd8-96c0-1757a4d74c37" path="/var/lib/kubelet/pods/e407743a-c859-4dd8-96c0-1757a4d74c37/volumes"
Oct 30 22:22:29.331099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:29.331070    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"
Oct 30 22:22:29.331517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:29.331142    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk" event={"ID":"261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47","Type":"ContainerDied","Data":"240eb66b544e8fe0e0a103ca40f7d8d397b0f1529ecd2eb2eabdb729aa7f5ddc"}
Oct 30 22:22:29.331517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:29.331197    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="240eb66b544e8fe0e0a103ca40f7d8d397b0f1529ecd2eb2eabdb729aa7f5ddc"
Oct 30 22:22:32.056934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.056889    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"]
Oct 30 22:22:32.104244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.104210    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1891ceae-752sh"]
Oct 30 22:22:32.352671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.352573    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:32.364343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.364309    5028 generic.go:356] "Generic (PLEG): container finished" podID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerID="a8aafca25e0a7ef81b53e6793456244aa30c49ea07b7fcc2a300997fc7da1ed0" exitCode=0
Oct 30 22:22:32.364498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.364382    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" event={"ID":"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b","Type":"ContainerDied","Data":"a8aafca25e0a7ef81b53e6793456244aa30c49ea07b7fcc2a300997fc7da1ed0"}
Oct 30 22:22:32.515668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.515641    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:22:32.636569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636492    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.636569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636556    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.636837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636662    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.636837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636794    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.636948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636841    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.636948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636898    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.637061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636948    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.637061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.636955    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:32.637061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637014    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.637218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637101    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token\") pod \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\" (UID: \"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b\") "
Oct 30 22:22:32.637218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637128    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:32.637218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637189    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:32.637380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637308    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:32.637840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637613    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:32.637961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637853    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.637961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637876    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.637961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637891    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.637961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637934    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.637961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.637947    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.640189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.640150    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "kube-api-access-v45z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:32.640987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.640958    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:32.641141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.640995    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:32.641795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.641772    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy" pod "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" (UID: "6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:32.739640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.739602    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.739640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.739635    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.739640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.739647    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:32.739888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:32.739656    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45z7\" (UniqueName: \"kubernetes.io/projected/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b-kube-api-access-v45z7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:33.295373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:33.295339    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d44fe5-c600-4c0c-8012-1d40c0c44346" path="/var/lib/kubelet/pods/f9d44fe5-c600-4c0c-8012-1d40c0c44346/volumes"
Oct 30 22:22:33.376873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:33.376845    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"
Oct 30 22:22:33.377043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:33.376848    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8" event={"ID":"6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b","Type":"ContainerDied","Data":"1bee575138aa0bb53305f8062785b9d2abf6c043ded6c9e88c1a7f8143dec8bf"}
Oct 30 22:22:33.377043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:33.376969    5028 scope.go:122] "RemoveContainer" containerID="a8aafca25e0a7ef81b53e6793456244aa30c49ea07b7fcc2a300997fc7da1ed0"
Oct 30 22:22:33.585351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:33.585268    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"]
Oct 30 22:22:33.632443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:33.632405    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdcn8"]
Oct 30 22:22:34.958678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.958638    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"]
Oct 30 22:22:34.959345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959330    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy"
Oct 30 22:22:34.959406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959346    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy"
Oct 30 22:22:34.959406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959362    5028 cpu_manager.go:414] "containerMap: removing container" podUID="84c0158a-8d75-4f4c-bfbf-7fea099c1192" containerName="revoke-keys"
Oct 30 22:22:34.959406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959367    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c0158a-8d75-4f4c-bfbf-7fea099c1192" containerName="revoke-keys"
Oct 30 22:22:34.959406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959385    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9947f535-998c-4fb2-b6e9-f98fae674a36" containerName="revoke-keys"
Oct 30 22:22:34.959406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959391    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9947f535-998c-4fb2-b6e9-f98fae674a36" containerName="revoke-keys"
Oct 30 22:22:34.959556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959414    5028 cpu_manager.go:414] "containerMap: removing container" podUID="261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" containerName="revoke-keys"
Oct 30 22:22:34.959556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959420    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" containerName="revoke-keys"
Oct 30 22:22:34.959556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959512    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9947f535-998c-4fb2-b6e9-f98fae674a36" containerName="revoke-keys"
Oct 30 22:22:34.959556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959525    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" containerName="revoke-keys"
Oct 30 22:22:34.959556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959534    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" containerName="istio-proxy"
Oct 30 22:22:34.959556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.959541    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c0158a-8d75-4f4c-bfbf-7fea099c1192" containerName="revoke-keys"
Oct 30 22:22:34.964850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:34.964823    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.017431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.017395    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"]
Oct 30 22:22:35.057705 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.057648    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:35.061895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.061869    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:35.061895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.061881    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:22:35.062056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.061922    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:35.072829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.072804    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.073034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.073015    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.073181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.073160    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46x2r\" (UniqueName: \"kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.174703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.174640    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.174890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.174862    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.174975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.174953    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46x2r\" (UniqueName: \"kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.175494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.175472    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.177229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.177200    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.234189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.234108    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46x2r\" (UniqueName: \"kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r\") pod \"maas-api-revoke-sub-6db53485-nc4kv\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.279217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.279182    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:35.292919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.292890    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b" path="/var/lib/kubelet/pods/6bbb79d5-a4a8-40c9-b12f-7f1c3ff9e56b/volumes"
Oct 30 22:22:35.594777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.594671    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"]
Oct 30 22:22:35.600670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.600640    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.681734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.681671    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.681734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.681740    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.681975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.681843    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.681975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.681926    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.681975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.681954    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.682101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.682063    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7g6l\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.682101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.682095    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.682205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.682117    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.682205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.682141    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.706739 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:35.698925    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda044fb0a_059b_4716_9b65_60deab08a2e5.slice/crio-019ca3f09366ef726c756aa6a4556b68f9c4a06de9fd3ff920c640bc5564a427 WatchSource:0}: Error finding container 019ca3f09366ef726c756aa6a4556b68f9c4a06de9fd3ff920c640bc5564a427: Status 404 returned error can't find the container with id 019ca3f09366ef726c756aa6a4556b68f9c4a06de9fd3ff920c640bc5564a427
Oct 30 22:22:35.710716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.710668    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"]
Oct 30 22:22:35.715358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.715329    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"]
Oct 30 22:22:35.783622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783585    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.783622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783630    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.783886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783668    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.783886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783755    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.783886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783782    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783887    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7g6l\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783937    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.783973    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.784001    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.784099    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.784212    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.784374    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.784502    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.784636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.784611    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.786503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.786474    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.786605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.786539    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.872425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.872339    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"]
Oct 30 22:22:35.878705 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.878667    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"]
Oct 30 22:22:35.879071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.878802    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:35.884326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.884298    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.918481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.918448    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.918602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.918532    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7g6l\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:35.929414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.929366    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"]
Oct 30 22:22:35.986944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.986907    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987056    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987110    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9js\" (UniqueName: \"kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:35.987373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987164    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:35.987373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987245    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987332    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987371    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987448    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987467    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987500    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:35.987616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987533    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.987616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.987559    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:35.996791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:35.996758    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"]
Oct 30 22:22:36.088394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088357    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.088394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088393    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088426    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088457    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088480    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088513    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088574    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088627    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9js\" (UniqueName: \"kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.088678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088674    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.089067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088771    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088866    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088869    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088904    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.088968    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.089301    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.089374    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.089531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.089489    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.089679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.089657    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.091308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.091285    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.091406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.091338    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.091445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.091405    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.148251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.148212    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.148368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.148270    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.148368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.148326    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9js\" (UniqueName: \"kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js\") pod \"maas-api-revoke-sub-755c74df-8vtzp\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.195795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.195760    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:36.203251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.203114    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:36.217174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.217141    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:36.416750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.416607    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv" event={"ID":"a044fb0a-059b-4716-9b65-60deab08a2e5","Type":"ContainerStarted","Data":"39a89abd2a43a8509854f5474c92842462bfcc3736ec0589c5e6b63a568fcfce"}
Oct 30 22:22:36.416750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.416681    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv" event={"ID":"a044fb0a-059b-4716-9b65-60deab08a2e5","Type":"ContainerStarted","Data":"019ca3f09366ef726c756aa6a4556b68f9c4a06de9fd3ff920c640bc5564a427"}
Oct 30 22:22:36.528240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.528186    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv" podStartSLOduration=2.5281677350000002 podStartE2EDuration="2.528167735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:36.527349811 +0000 UTC m=+1287.376862784" watchObservedRunningTime="2026-10-30 22:22:36.528167735 +0000 UTC m=+1287.377680712"
Oct 30 22:22:36.579921 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:36.579881    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e589c9e_80fa_46d8_a78d_bd60e2120611.slice/crio-322dbdfb80fd54dffe492a4fae52c5a0a0777d9705b567eec2425c739525cceb WatchSource:0}: Error finding container 322dbdfb80fd54dffe492a4fae52c5a0a0777d9705b567eec2425c739525cceb: Status 404 returned error can't find the container with id 322dbdfb80fd54dffe492a4fae52c5a0a0777d9705b567eec2425c739525cceb
Oct 30 22:22:36.581028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.581002    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"]
Oct 30 22:22:36.630872 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:36.630842    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720d702d_4c02_44e6_892d_6d55d71092e5.slice/crio-daa0da751014fe5d31062a43311ffb739dcb54b644966dc58c8a7e451ab79d90 WatchSource:0}: Error finding container daa0da751014fe5d31062a43311ffb739dcb54b644966dc58c8a7e451ab79d90: Status 404 returned error can't find the container with id daa0da751014fe5d31062a43311ffb739dcb54b644966dc58c8a7e451ab79d90
Oct 30 22:22:36.633588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.633557    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"]
Oct 30 22:22:36.634007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.633974    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:36.634110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.634049    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:36.634110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.634095    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:36.669608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.669579    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"]
Oct 30 22:22:36.670188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.670155    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:36.670282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.670233    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:36.670354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.670279    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:36.926414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:36.926322    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:37.077523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.077486    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"]
Oct 30 22:22:37.082594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.082564    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3577d5d-lwwld"]
Oct 30 22:22:37.296853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.296821    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b085ea-e1dc-4573-b9a5-344f09d0d863" path="/var/lib/kubelet/pods/b6b085ea-e1dc-4573-b9a5-344f09d0d863/volumes"
Oct 30 22:22:37.430612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.430556    5028 generic.go:356] "Generic (PLEG): container finished" podID="a044fb0a-059b-4716-9b65-60deab08a2e5" containerID="39a89abd2a43a8509854f5474c92842462bfcc3736ec0589c5e6b63a568fcfce" exitCode=0
Oct 30 22:22:37.430819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.430730    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv" event={"ID":"a044fb0a-059b-4716-9b65-60deab08a2e5","Type":"ContainerDied","Data":"39a89abd2a43a8509854f5474c92842462bfcc3736ec0589c5e6b63a568fcfce"}
Oct 30 22:22:37.432967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.432940    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" event={"ID":"720d702d-4c02-44e6-892d-6d55d71092e5","Type":"ContainerStarted","Data":"0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f"}
Oct 30 22:22:37.433086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.432992    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" event={"ID":"720d702d-4c02-44e6-892d-6d55d71092e5","Type":"ContainerStarted","Data":"daa0da751014fe5d31062a43311ffb739dcb54b644966dc58c8a7e451ab79d90"}
Oct 30 22:22:37.434886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.434858    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" event={"ID":"2f1933fb-6dac-4791-89cc-04fabc2aaaa3","Type":"ContainerStarted","Data":"3e4f64f74329ce9f3392af8e44ef7c4b5c8f1cf1cf835f7484b07654b87e8e11"}
Oct 30 22:22:37.434995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.434916    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" event={"ID":"2f1933fb-6dac-4791-89cc-04fabc2aaaa3","Type":"ContainerStarted","Data":"49ad1b05552fd82d54eb369e3a9a0660b2956671f79a1a59e929e824922a63b5"}
Oct 30 22:22:37.436737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.436713    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp" event={"ID":"8e589c9e-80fa-46d8-a78d-bd60e2120611","Type":"ContainerStarted","Data":"73b811f043eaced974e137fe4b79d99158494893c6c2bb81f8ef7cf4bdf3d301"}
Oct 30 22:22:37.436856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.436756    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp" event={"ID":"8e589c9e-80fa-46d8-a78d-bd60e2120611","Type":"ContainerStarted","Data":"322dbdfb80fd54dffe492a4fae52c5a0a0777d9705b567eec2425c739525cceb"}
Oct 30 22:22:37.664446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.664386    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podStartSLOduration=2.664371788 podStartE2EDuration="2.664371788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.662097597 +0000 UTC m=+1288.511610582" watchObservedRunningTime="2026-10-30 22:22:37.664371788 +0000 UTC m=+1288.513884824"
Oct 30 22:22:37.717247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.717208    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"]
Oct 30 22:22:37.722483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.722458    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.786165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.786117    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"]
Oct 30 22:22:37.795604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.795530    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp" podStartSLOduration=2.79550488 podStartE2EDuration="2.79550488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.777634307 +0000 UTC m=+1288.627147281" watchObservedRunningTime="2026-10-30 22:22:37.79550488 +0000 UTC m=+1288.645017854"
Oct 30 22:22:37.825710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.822790    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4ph\" (UniqueName: \"kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.825710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.822852    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.825710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.822944    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.984909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.984866    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4ph\" (UniqueName: \"kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.985190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.985170    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.985418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.985393    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.986747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.986722    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.990409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.989838    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:37.994182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:37.994138    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podStartSLOduration=2.994126482 podStartE2EDuration="2.994126482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.99005493 +0000 UTC m=+1288.839567905" watchObservedRunningTime="2026-10-30 22:22:37.994126482 +0000 UTC m=+1288.843639476"
Oct 30 22:22:38.063840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.063803    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4ph\" (UniqueName: \"kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph\") pod \"maas-api-revoke-sub-e8e4c92f-cgvnf\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:38.203588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.203546    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:38.205264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.205243    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:38.205322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.205292    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:38.225496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.225405    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:38.243806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.243759    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:38.244131 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.244101    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:38.344986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.344949    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:38.502932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.502847    5028 generic.go:356] "Generic (PLEG): container finished" podID="8e589c9e-80fa-46d8-a78d-bd60e2120611" containerID="73b811f043eaced974e137fe4b79d99158494893c6c2bb81f8ef7cf4bdf3d301" exitCode=0
Oct 30 22:22:38.503765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.503729    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp" event={"ID":"8e589c9e-80fa-46d8-a78d-bd60e2120611","Type":"ContainerDied","Data":"73b811f043eaced974e137fe4b79d99158494893c6c2bb81f8ef7cf4bdf3d301"}
Oct 30 22:22:38.915250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:38.913829    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"]
Oct 30 22:22:39.093095 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.093022    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"]
Oct 30 22:22:39.095366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.095058    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:39.099015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.098988    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed82e841-lhx2r"]
Oct 30 22:22:39.141711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.141578    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca\") pod \"a044fb0a-059b-4716-9b65-60deab08a2e5\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") "
Oct 30 22:22:39.141711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.141656    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca\") pod \"a044fb0a-059b-4716-9b65-60deab08a2e5\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") "
Oct 30 22:22:39.141711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.141719    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r\" (UniqueName: \"kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r\") pod \"a044fb0a-059b-4716-9b65-60deab08a2e5\" (UID: \"a044fb0a-059b-4716-9b65-60deab08a2e5\") "
Oct 30 22:22:39.188008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.187961    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca" pod "a044fb0a-059b-4716-9b65-60deab08a2e5" (UID: "a044fb0a-059b-4716-9b65-60deab08a2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:39.189733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.188894    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca" pod "a044fb0a-059b-4716-9b65-60deab08a2e5" (UID: "a044fb0a-059b-4716-9b65-60deab08a2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:39.193277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.193193    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r" pod "a044fb0a-059b-4716-9b65-60deab08a2e5" (UID: "a044fb0a-059b-4716-9b65-60deab08a2e5"). InnerVolumeSpecName "kube-api-access-46x2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:39.204190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.204156    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.204526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.204221    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:39.245169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.245135    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.245465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.245424    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:39.290788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.289805    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:39.290788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.289842    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a044fb0a-059b-4716-9b65-60deab08a2e5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:39.290788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.289860    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46x2r\" (UniqueName: \"kubernetes.io/projected/a044fb0a-059b-4716-9b65-60deab08a2e5-kube-api-access-46x2r\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:39.303634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.302628    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf6eca3-c77f-45cc-b33a-25c4addcf3b6" path="/var/lib/kubelet/pods/0cf6eca3-c77f-45cc-b33a-25c4addcf3b6/volumes"
Oct 30 22:22:39.520738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.516775    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv" event={"ID":"a044fb0a-059b-4716-9b65-60deab08a2e5","Type":"ContainerDied","Data":"019ca3f09366ef726c756aa6a4556b68f9c4a06de9fd3ff920c640bc5564a427"}
Oct 30 22:22:39.522114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.522084    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019ca3f09366ef726c756aa6a4556b68f9c4a06de9fd3ff920c640bc5564a427"
Oct 30 22:22:39.522336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.522320    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"
Oct 30 22:22:39.538713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.532126    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf" event={"ID":"190a8ef4-20c7-466f-83ed-f0afa1d14370","Type":"ContainerStarted","Data":"21709f60bf2ba413a679d66a52c7c5afd07a355c43b90d0c73f277d72f9ccac2"}
Oct 30 22:22:39.996730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:39.996636    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:40.001918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.001897    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js\" (UniqueName: \"kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js\") pod \"8e589c9e-80fa-46d8-a78d-bd60e2120611\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") "
Oct 30 22:22:40.002061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.002050    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca\") pod \"8e589c9e-80fa-46d8-a78d-bd60e2120611\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") "
Oct 30 22:22:40.002137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.002125    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca\") pod \"8e589c9e-80fa-46d8-a78d-bd60e2120611\" (UID: \"8e589c9e-80fa-46d8-a78d-bd60e2120611\") "
Oct 30 22:22:40.003181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.003159    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca" pod "8e589c9e-80fa-46d8-a78d-bd60e2120611" (UID: "8e589c9e-80fa-46d8-a78d-bd60e2120611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:40.004795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.004774    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca" pod "8e589c9e-80fa-46d8-a78d-bd60e2120611" (UID: "8e589c9e-80fa-46d8-a78d-bd60e2120611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:40.004909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.004839    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js" pod "8e589c9e-80fa-46d8-a78d-bd60e2120611" (UID: "8e589c9e-80fa-46d8-a78d-bd60e2120611"). InnerVolumeSpecName "kube-api-access-kq9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:40.104051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.104014    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:40.104051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.104045    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e589c9e-80fa-46d8-a78d-bd60e2120611-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:40.104051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.104055    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9js\" (UniqueName: \"kubernetes.io/projected/8e589c9e-80fa-46d8-a78d-bd60e2120611-kube-api-access-kq9js\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:40.203994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.203961    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.204167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.204048    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:40.286771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.286640    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.286771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.286729    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:40.587872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.587730    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp" event={"ID":"8e589c9e-80fa-46d8-a78d-bd60e2120611","Type":"ContainerDied","Data":"322dbdfb80fd54dffe492a4fae52c5a0a0777d9705b567eec2425c739525cceb"}
Oct 30 22:22:40.587872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.587795    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="322dbdfb80fd54dffe492a4fae52c5a0a0777d9705b567eec2425c739525cceb"
Oct 30 22:22:40.588502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.588481    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"
Oct 30 22:22:40.590409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.590114    5028 generic.go:356] "Generic (PLEG): container finished" podID="190a8ef4-20c7-466f-83ed-f0afa1d14370" containerID="520ca174e7c20f130192afd6808ab6f3ab13bc56fbb5e169bc00bf230e6ae79c" exitCode=0
Oct 30 22:22:40.590409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:40.590150    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf" event={"ID":"190a8ef4-20c7-466f-83ed-f0afa1d14370","Type":"ContainerDied","Data":"520ca174e7c20f130192afd6808ab6f3ab13bc56fbb5e169bc00bf230e6ae79c"}
Oct 30 22:22:41.203673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:41.203639    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.203901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:41.203713    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:41.235713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:41.228296    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.260714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:41.246761    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:41.808267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:41.808163    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:42.100354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.100323    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:42.113350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.113320    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph\" (UniqueName: \"kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph\") pod \"190a8ef4-20c7-466f-83ed-f0afa1d14370\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") "
Oct 30 22:22:42.113511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.113489    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca\") pod \"190a8ef4-20c7-466f-83ed-f0afa1d14370\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") "
Oct 30 22:22:42.113741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.113723    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca\") pod \"190a8ef4-20c7-466f-83ed-f0afa1d14370\" (UID: \"190a8ef4-20c7-466f-83ed-f0afa1d14370\") "
Oct 30 22:22:42.114062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.114039    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca" pod "190a8ef4-20c7-466f-83ed-f0afa1d14370" (UID: "190a8ef4-20c7-466f-83ed-f0afa1d14370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:42.114151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.114133    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:42.115486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.115465    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph" pod "190a8ef4-20c7-466f-83ed-f0afa1d14370" (UID: "190a8ef4-20c7-466f-83ed-f0afa1d14370"). InnerVolumeSpecName "kube-api-access-fk4ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:42.115831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.115808    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca" pod "190a8ef4-20c7-466f-83ed-f0afa1d14370" (UID: "190a8ef4-20c7-466f-83ed-f0afa1d14370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:42.203719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.203667    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.203962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.203931    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:42.215868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.215799    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4ph\" (UniqueName: \"kubernetes.io/projected/190a8ef4-20c7-466f-83ed-f0afa1d14370-kube-api-access-fk4ph\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:42.215868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.215825    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190a8ef4-20c7-466f-83ed-f0afa1d14370-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:42.240125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.240086    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.240409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.240378    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:42.609412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.609335    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"
Oct 30 22:22:42.609412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.609376    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf" event={"ID":"190a8ef4-20c7-466f-83ed-f0afa1d14370","Type":"ContainerDied","Data":"21709f60bf2ba413a679d66a52c7c5afd07a355c43b90d0c73f277d72f9ccac2"}
Oct 30 22:22:42.609594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:42.609437    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21709f60bf2ba413a679d66a52c7c5afd07a355c43b90d0c73f277d72f9ccac2"
Oct 30 22:22:43.079708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.078785    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"]
Oct 30 22:22:43.085518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.085488    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9437dd04-wxczd"]
Oct 30 22:22:43.204020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.203988    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.204181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.204041    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:43.228714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.221973    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.228714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.222043    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:43.298388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:43.298354    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a24cae-60e8-4f64-8db0-296929f4cb92" path="/var/lib/kubelet/pods/a3a24cae-60e8-4f64-8db0-296929f4cb92/volumes"
Oct 30 22:22:44.055016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:44.054981    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"]
Oct 30 22:22:44.062854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:44.062822    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587e4216-jdtht"]
Oct 30 22:22:44.203870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:44.203833    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.204262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:44.203900    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 22:22:44.218403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:44.218355    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.218564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:44.218421    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 22:22:45.208636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.208603    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:45.209151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.209130    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:45.209855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.209840    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:22:45.223606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.223579    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:45.224027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.224002    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:45.224830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.224805    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:22:45.297672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:45.297635    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275f769a-1442-46b7-b905-b88c56205c49" path="/var/lib/kubelet/pods/275f769a-1442-46b7-b905-b88c56205c49/volumes"
Oct 30 22:22:46.200024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.199991    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"]
Oct 30 22:22:46.200827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.200804    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a044fb0a-059b-4716-9b65-60deab08a2e5" containerName="revoke-keys"
Oct 30 22:22:46.200827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.200827    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a044fb0a-059b-4716-9b65-60deab08a2e5" containerName="revoke-keys"
Oct 30 22:22:46.200999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.200869    5028 cpu_manager.go:414] "containerMap: removing container" podUID="190a8ef4-20c7-466f-83ed-f0afa1d14370" containerName="revoke-keys"
Oct 30 22:22:46.200999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.200878    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190a8ef4-20c7-466f-83ed-f0afa1d14370" containerName="revoke-keys"
Oct 30 22:22:46.200999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.200891    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8e589c9e-80fa-46d8-a78d-bd60e2120611" containerName="revoke-keys"
Oct 30 22:22:46.200999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.200900    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e589c9e-80fa-46d8-a78d-bd60e2120611" containerName="revoke-keys"
Oct 30 22:22:46.201211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.201058    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a044fb0a-059b-4716-9b65-60deab08a2e5" containerName="revoke-keys"
Oct 30 22:22:46.201211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.201076    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="190a8ef4-20c7-466f-83ed-f0afa1d14370" containerName="revoke-keys"
Oct 30 22:22:46.201211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.201086    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e589c9e-80fa-46d8-a78d-bd60e2120611" containerName="revoke-keys"
Oct 30 22:22:46.206186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.206162    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.246715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.246665    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.247058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.246839    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.247058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.246885    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mc4q\" (UniqueName: \"kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.257333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.257291    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:46.257333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.257307    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:46.257530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.257511    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:46.258981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.258958    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"]
Oct 30 22:22:46.300224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.300190    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:22:46.348346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.348312    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.348541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.348405    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.348541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.348435    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mc4q\" (UniqueName: \"kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.349079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.349054    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.350808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.350784    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.407366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.407321    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mc4q\" (UniqueName: \"kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q\") pod \"maas-api-revoke-sub-1ba7014e-d26zb\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.520964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.520930    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:46.580356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.580187    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:46.869744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.869638    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"]
Oct 30 22:22:46.876105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.876072    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:46.933571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.933513    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"]
Oct 30 22:22:46.983604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.983558    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"]
Oct 30 22:22:46.993645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.993173    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:46.993645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.993233    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:46.993645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:46.993510    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptng\" (UniqueName: \"kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.095073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.095033    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptng\" (UniqueName: \"kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.095254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.095153    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.095254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.095199    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.096032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.096005    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.097713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.097657    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.155096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.155010    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptng\" (UniqueName: \"kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng\") pod \"maas-api-revoke-sub-4f5ccb7f-jx79m\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:47.221727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:47.221677    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:48.556708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:48.555015    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"]
Oct 30 22:22:48.730242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:48.730155    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb" event={"ID":"4051c196-a060-49aa-a286-90208a51848a","Type":"ContainerStarted","Data":"147eda251f68ae597c51158c4edd59e9e4a5898ba478b12ad017e74b8fda67bf"}
Oct 30 22:22:48.730518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:48.730483    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb" event={"ID":"4051c196-a060-49aa-a286-90208a51848a","Type":"ContainerStarted","Data":"18e728eda033f45a4c5d064f85ec5c6558bffad1cb3c63314a3884a15d55742c"}
Oct 30 22:22:48.945711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:48.943602    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb" podStartSLOduration=2.943581399 podStartE2EDuration="2.943581399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:48.906332376 +0000 UTC m=+1299.755845350" watchObservedRunningTime="2026-10-30 22:22:48.943581399 +0000 UTC m=+1299.793094372"
Oct 30 22:22:49.036906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:49.036819    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:49.248761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:49.246041    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:49.311952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:49.311780    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:49.428722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:49.420018    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:49.855770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:49.854986    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.052345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.044329    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.052345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.045014    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:22:50.054310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.052725    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.339715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.329536    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:50.339715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.330664    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:22:50.340697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.340645    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.359092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.359061    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:22:50.427707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.361132    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.427707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.361364    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:22:50.427707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.423072    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.427707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.423519    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:22:50.936171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.904584    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:50.936171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.905120    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:22:50.942713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.939463    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m" event={"ID":"66d5dcbe-e6e7-44ec-abe8-b2141132666a","Type":"ContainerStarted","Data":"8265890a7ce07fee6d98b4c8316245ac1822a9970c656e76d3c9df9bb9a4ecfa"}
Oct 30 22:22:50.972443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.972406    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m" event={"ID":"66d5dcbe-e6e7-44ec-abe8-b2141132666a","Type":"ContainerStarted","Data":"8c7f73256ead8566568e100ebb94471fa0ec659801eab78802a620d3d22f5137"}
Oct 30 22:22:50.977480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.977324    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:50.977781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:50.977751    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:51.317126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.033668    5028 generic.go:356] "Generic (PLEG): container finished" podID="4051c196-a060-49aa-a286-90208a51848a" containerID="147eda251f68ae597c51158c4edd59e9e4a5898ba478b12ad017e74b8fda67bf" exitCode=0
Oct 30 22:22:51.391714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.033718    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb" event={"ID":"4051c196-a060-49aa-a286-90208a51848a","Type":"ContainerDied","Data":"147eda251f68ae597c51158c4edd59e9e4a5898ba478b12ad017e74b8fda67bf"}
Oct 30 22:22:51.455713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.054813    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:51.456049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.095647    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:22:51.456949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.148595    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:51.457136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.363830    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:51.482726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.363863    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:22:51.482726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.364361    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:51.482726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.364388    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:51.482726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.426234    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:51.933842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:51.933793    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:52.160716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.114674    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:22:52.161807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.161777    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:22:52.162039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.162024    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:22:52.186451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.186363    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:22:52.498708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.496671    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:52.498708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.497040    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:22:52.498708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.497311    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:52.498708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.497592    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:22:52.498708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.497853    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:22:52.498708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.498129    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:22:52.738727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.733012    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"]
Oct 30 22:22:52.781812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.761151    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"]
Oct 30 22:22:52.783957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.761324    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.821933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.821860    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m" podStartSLOduration=6.821839558 podStartE2EDuration="6.821839558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:51.806310908 +0000 UTC m=+1302.655823885" watchObservedRunningTime="2026-10-30 22:22:52.821839558 +0000 UTC m=+1303.671352533"
Oct 30 22:22:52.977646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.977610    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:22:52.979480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.979197    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.992219    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994264    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994341    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994437    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994474    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994522    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994570    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994618    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.994980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.994754    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.996908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.996679    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:22:52.997528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.997478    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqrg\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:52.998702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.998659    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:22:53.001711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:52.999781    5028 generic.go:356] "Generic (PLEG): container finished" podID="66d5dcbe-e6e7-44ec-abe8-b2141132666a" containerID="8265890a7ce07fee6d98b4c8316245ac1822a9970c656e76d3c9df9bb9a4ecfa" exitCode=0
Oct 30 22:22:53.001711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.000341    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m" event={"ID":"66d5dcbe-e6e7-44ec-abe8-b2141132666a","Type":"ContainerDied","Data":"8265890a7ce07fee6d98b4c8316245ac1822a9970c656e76d3c9df9bb9a4ecfa"}
Oct 30 22:22:53.019182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.018964    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:22:53.066523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.066436    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:22:53.098621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.098586    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.098837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.098799    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.098939    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.098989    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.099084    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.099181    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.099230    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.099417    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.099720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.099467    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqrg\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.100197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.099924    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.101873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.100432    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.101873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.101198    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.101873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.101446    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.101873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.101843    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.102797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.102779    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.105427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.104421    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.169608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.169575    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.175393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.175256    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqrg\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.255313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.255282    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:53.282705 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.282666    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:53.302610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.302581    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca\") pod \"4051c196-a060-49aa-a286-90208a51848a\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") "
Oct 30 22:22:53.302763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.302676    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q\" (UniqueName: \"kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q\") pod \"4051c196-a060-49aa-a286-90208a51848a\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") "
Oct 30 22:22:53.303025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.302986    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca\") pod \"4051c196-a060-49aa-a286-90208a51848a\" (UID: \"4051c196-a060-49aa-a286-90208a51848a\") "
Oct 30 22:22:53.303425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.303401    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca" pod "4051c196-a060-49aa-a286-90208a51848a" (UID: "4051c196-a060-49aa-a286-90208a51848a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.304891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.304862    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q" pod "4051c196-a060-49aa-a286-90208a51848a" (UID: "4051c196-a060-49aa-a286-90208a51848a"). InnerVolumeSpecName "kube-api-access-8mc4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.305079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.305055    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca" pod "4051c196-a060-49aa-a286-90208a51848a" (UID: "4051c196-a060-49aa-a286-90208a51848a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:53.327399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.327323    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:53.411815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.404965    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4051c196-a060-49aa-a286-90208a51848a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.411815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.404995    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4051c196-a060-49aa-a286-90208a51848a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.411815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.405010    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mc4q\" (UniqueName: \"kubernetes.io/projected/4051c196-a060-49aa-a286-90208a51848a-kube-api-access-8mc4q\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.776376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.776293    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"]
Oct 30 22:22:53.780872 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:53.780844    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7871f4_4a43_42cd_ae4a_33e3f89cd26a.slice/crio-0f20611e6db03448948435323c85dedbfd3c97934dffce80f8cac4f6a2dc63f8 WatchSource:0}: Error finding container 0f20611e6db03448948435323c85dedbfd3c97934dffce80f8cac4f6a2dc63f8: Status 404 returned error can't find the container with id 0f20611e6db03448948435323c85dedbfd3c97934dffce80f8cac4f6a2dc63f8
Oct 30 22:22:53.784323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.784285    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:53.784442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.784367    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:53.784442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:53.784409    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:22:54.019294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.019256    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" event={"ID":"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a","Type":"ContainerStarted","Data":"1c1619c85c55dd42a4aa3a0e90722d9d61b5038de54ff1537f1848bd7a4b4ba0"}
Oct 30 22:22:54.019565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.019544    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" event={"ID":"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a","Type":"ContainerStarted","Data":"0f20611e6db03448948435323c85dedbfd3c97934dffce80f8cac4f6a2dc63f8"}
Oct 30 22:22:54.026773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.022579    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"
Oct 30 22:22:54.026773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.024362    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb" event={"ID":"4051c196-a060-49aa-a286-90208a51848a","Type":"ContainerDied","Data":"18e728eda033f45a4c5d064f85ec5c6558bffad1cb3c63314a3884a15d55742c"}
Oct 30 22:22:54.026773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.024409    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e728eda033f45a4c5d064f85ec5c6558bffad1cb3c63314a3884a15d55742c"
Oct 30 22:22:54.274744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.271727    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:54.327591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.327508    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:22:54.329103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.329073    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:22:54.329253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.329131    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:22:54.332673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.332645    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca\") pod \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") "
Oct 30 22:22:54.332804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.332786    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca\") pod \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") "
Oct 30 22:22:54.332898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.332882    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng\" (UniqueName: \"kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng\") pod \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\" (UID: \"66d5dcbe-e6e7-44ec-abe8-b2141132666a\") "
Oct 30 22:22:54.333226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.333185    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca" pod "66d5dcbe-e6e7-44ec-abe8-b2141132666a" (UID: "66d5dcbe-e6e7-44ec-abe8-b2141132666a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:54.334247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.334115    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.339734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.335471    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca" pod "66d5dcbe-e6e7-44ec-abe8-b2141132666a" (UID: "66d5dcbe-e6e7-44ec-abe8-b2141132666a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:54.339734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.335584    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng" pod "66d5dcbe-e6e7-44ec-abe8-b2141132666a" (UID: "66d5dcbe-e6e7-44ec-abe8-b2141132666a"). InnerVolumeSpecName "kube-api-access-7ptng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:54.378323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.378266    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podStartSLOduration=4.378251625 podStartE2EDuration="4.378251625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:54.37448421 +0000 UTC m=+1305.223997184" watchObservedRunningTime="2026-10-30 22:22:54.378251625 +0000 UTC m=+1305.227764597"
Oct 30 22:22:54.435839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.435802    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d5dcbe-e6e7-44ec-abe8-b2141132666a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.435839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.435838    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptng\" (UniqueName: \"kubernetes.io/projected/66d5dcbe-e6e7-44ec-abe8-b2141132666a-kube-api-access-7ptng\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.994355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.994317    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"]
Oct 30 22:22:54.995118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.995097    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4051c196-a060-49aa-a286-90208a51848a" containerName="revoke-keys"
Oct 30 22:22:54.995229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.995125    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4051c196-a060-49aa-a286-90208a51848a" containerName="revoke-keys"
Oct 30 22:22:54.995229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.995165    5028 cpu_manager.go:414] "containerMap: removing container" podUID="66d5dcbe-e6e7-44ec-abe8-b2141132666a" containerName="revoke-keys"
Oct 30 22:22:54.995229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.995174    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d5dcbe-e6e7-44ec-abe8-b2141132666a" containerName="revoke-keys"
Oct 30 22:22:54.995416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.995376    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4051c196-a060-49aa-a286-90208a51848a" containerName="revoke-keys"
Oct 30 22:22:54.995416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:54.995398    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d5dcbe-e6e7-44ec-abe8-b2141132666a" containerName="revoke-keys"
Oct 30 22:22:55.003786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.002301    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.071602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.071121    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"]
Oct 30 22:22:55.073828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.072094    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnxb\" (UniqueName: \"kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.073828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.072194    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.073828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.072424    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.083354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.080730    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"
Oct 30 22:22:55.084789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.084758    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m" event={"ID":"66d5dcbe-e6e7-44ec-abe8-b2141132666a","Type":"ContainerDied","Data":"8c7f73256ead8566568e100ebb94471fa0ec659801eab78802a620d3d22f5137"}
Oct 30 22:22:55.084972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.084958    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7f73256ead8566568e100ebb94471fa0ec659801eab78802a620d3d22f5137"
Oct 30 22:22:55.085060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.085050    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"]
Oct 30 22:22:55.091182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.091158    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ffce53-lxmfl"]
Oct 30 22:22:55.173753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.173676    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.174257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.174227    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.174642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.174379    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.174799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.174622    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnxb\" (UniqueName: \"kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.177169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.177143    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.249929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.249829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnxb\" (UniqueName: \"kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb\") pod \"maas-api-revoke-sub-b77334a6-2bs7t\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.292925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.292892    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c172f1-c67c-4cfc-a481-ffbba5c84c95" path="/var/lib/kubelet/pods/30c172f1-c67c-4cfc-a481-ffbba5c84c95/volumes"
Oct 30 22:22:55.336712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.335122    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:22:55.336712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.335201    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:22:55.359113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.359075    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:55.765250 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:55.765213    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f9b0cc_326e_429d_9a32_1694cafc92a8.slice/crio-09891e21e0cb9ded2134b129537c77d17fc42a1a1ddd21758c5e67551053e41b WatchSource:0}: Error finding container 09891e21e0cb9ded2134b129537c77d17fc42a1a1ddd21758c5e67551053e41b: Status 404 returned error can't find the container with id 09891e21e0cb9ded2134b129537c77d17fc42a1a1ddd21758c5e67551053e41b
Oct 30 22:22:55.768806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:55.768778    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"]
Oct 30 22:22:56.090958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.090862    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t" event={"ID":"d0f9b0cc-326e-429d-9a32-1694cafc92a8","Type":"ContainerStarted","Data":"09891e21e0cb9ded2134b129537c77d17fc42a1a1ddd21758c5e67551053e41b"}
Oct 30 22:22:56.338710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.335130    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:22:56.338710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.335191    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:22:56.570046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.570008    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:56.725711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.725294    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:56.959612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.959575    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"]
Oct 30 22:22:56.964295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:56.964270    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.005400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.005367    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"]
Oct 30 22:22:57.103940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.103795    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.103940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.103873    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxw9\" (UniqueName: \"kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.103940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.103941    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.148104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.146523    5028 generic.go:356] "Generic (PLEG): container finished" podID="d0f9b0cc-326e-429d-9a32-1694cafc92a8" containerID="bf63dbc510d1050357b5ffef0ec0953e1df49da1bafd2b6f6508a2857dd71e7e" exitCode=0
Oct 30 22:22:57.148104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.146744    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t" event={"ID":"d0f9b0cc-326e-429d-9a32-1694cafc92a8","Type":"ContainerDied","Data":"bf63dbc510d1050357b5ffef0ec0953e1df49da1bafd2b6f6508a2857dd71e7e"}
Oct 30 22:22:57.205510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.205425    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.205510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.205487    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxw9\" (UniqueName: \"kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.205728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.205531    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.206249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.206229    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.207889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.207869    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.267551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.267513    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxw9\" (UniqueName: \"kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9\") pod \"maas-api-revoke-sub-0195b687-nfnrl\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.333712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.328549    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:22:57.333712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.328611    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:22:57.342712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.342248    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:22:57.555809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.555766    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"]
Oct 30 22:22:57.577788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.577752    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.583288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.583252    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"]
Oct 30 22:22:57.593578 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:22:57.593548    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bc7ef2_5e28_4e1a_89b7_d607394c017e.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:22:57.713139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.713103    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmxr\" (UniqueName: \"kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.713325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.713263    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.713408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.713382    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.757088 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:57.755816    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f8add_e6c4_41c2_aa95_fae5695b1e61.slice/crio-44c2a1f7e9f71642937035b2b96e148f401a61ca4d0fdeb7b090d8179d233a02 WatchSource:0}: Error finding container 44c2a1f7e9f71642937035b2b96e148f401a61ca4d0fdeb7b090d8179d233a02: Status 404 returned error can't find the container with id 44c2a1f7e9f71642937035b2b96e148f401a61ca4d0fdeb7b090d8179d233a02
Oct 30 22:22:57.757356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.757331    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"]
Oct 30 22:22:57.814941 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.814853    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmxr\" (UniqueName: \"kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.815108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.815022    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.815108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.815086    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.815726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.815677    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.850317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.850242    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:57.903323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.903286    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.933795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:57.929468    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmxr\" (UniqueName: \"kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr\") pod \"maas-api-revoke-sub-7f3b5eaf-sjnkz\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:58.165848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.165818    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl" event={"ID":"3b2f8add-e6c4-41c2-aa95-fae5695b1e61","Type":"ContainerStarted","Data":"44c2a1f7e9f71642937035b2b96e148f401a61ca4d0fdeb7b090d8179d233a02"}
Oct 30 22:22:58.198117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.198079    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:22:58.358719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.348407    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:22:58.358719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.348464    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:22:58.480987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.480953    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:58.549942 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:22:58.549893    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:22:58.580143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.579846    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca\") pod \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") "
Oct 30 22:22:58.580143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.580082    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca\") pod \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") "
Oct 30 22:22:58.580143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.580118    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb\" (UniqueName: \"kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb\") pod \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\" (UID: \"d0f9b0cc-326e-429d-9a32-1694cafc92a8\") "
Oct 30 22:22:58.581593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.580601    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca" pod "d0f9b0cc-326e-429d-9a32-1694cafc92a8" (UID: "d0f9b0cc-326e-429d-9a32-1694cafc92a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:58.584026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.583976    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca" pod "d0f9b0cc-326e-429d-9a32-1694cafc92a8" (UID: "d0f9b0cc-326e-429d-9a32-1694cafc92a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:58.584026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.583990    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb" pod "d0f9b0cc-326e-429d-9a32-1694cafc92a8" (UID: "d0f9b0cc-326e-429d-9a32-1694cafc92a8"). InnerVolumeSpecName "kube-api-access-rnnxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:58.681839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.681809    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.681839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.681837    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnnxb\" (UniqueName: \"kubernetes.io/projected/d0f9b0cc-326e-429d-9a32-1694cafc92a8-kube-api-access-rnnxb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.681839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.681847    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f9b0cc-326e-429d-9a32-1694cafc92a8-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.691082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:58.691046    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"]
Oct 30 22:22:58.701028 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:22:58.700992    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bc7ef2_5e28_4e1a_89b7_d607394c017e.slice/crio-6dd2b9cf8df78c8d87a8b0f5b135bbc4bfd04d81e21cde9178ab7f3971b1c7a4 WatchSource:0}: Error finding container 6dd2b9cf8df78c8d87a8b0f5b135bbc4bfd04d81e21cde9178ab7f3971b1c7a4: Status 404 returned error can't find the container with id 6dd2b9cf8df78c8d87a8b0f5b135bbc4bfd04d81e21cde9178ab7f3971b1c7a4
Oct 30 22:22:59.065056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.064954    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"]
Oct 30 22:22:59.071810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.071778    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036988bc-vjkqn"]
Oct 30 22:22:59.179405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.179370    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz" event={"ID":"11bc7ef2-5e28-4e1a-89b7-d607394c017e","Type":"ContainerStarted","Data":"46f42031ea491b78e9f53d48983691a44b1af29fec04a5ab59a6cd5b8ea6283c"}
Oct 30 22:22:59.179563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.179425    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz" event={"ID":"11bc7ef2-5e28-4e1a-89b7-d607394c017e","Type":"ContainerStarted","Data":"6dd2b9cf8df78c8d87a8b0f5b135bbc4bfd04d81e21cde9178ab7f3971b1c7a4"}
Oct 30 22:22:59.181231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.181206    5028 generic.go:356] "Generic (PLEG): container finished" podID="3b2f8add-e6c4-41c2-aa95-fae5695b1e61" containerID="6e9d4ed333e2f3d725b41cc6dc8d95e7bb3edad9787d51f112e20e4974d4205b" exitCode=0
Oct 30 22:22:59.181440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.181426    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl" event={"ID":"3b2f8add-e6c4-41c2-aa95-fae5695b1e61","Type":"ContainerDied","Data":"6e9d4ed333e2f3d725b41cc6dc8d95e7bb3edad9787d51f112e20e4974d4205b"}
Oct 30 22:22:59.183006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.182987    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t" event={"ID":"d0f9b0cc-326e-429d-9a32-1694cafc92a8","Type":"ContainerDied","Data":"09891e21e0cb9ded2134b129537c77d17fc42a1a1ddd21758c5e67551053e41b"}
Oct 30 22:22:59.183146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.183129    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09891e21e0cb9ded2134b129537c77d17fc42a1a1ddd21758c5e67551053e41b"
Oct 30 22:22:59.183146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.183016    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"
Oct 30 22:22:59.295952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.295909    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae4ee13-fe7d-4b1b-bbc0-23c625fed66d" path="/var/lib/kubelet/pods/dae4ee13-fe7d-4b1b-bbc0-23c625fed66d/volumes"
Oct 30 22:22:59.303640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.303579    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz" podStartSLOduration=2.303558517 podStartE2EDuration="2.303558517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:59.296766555 +0000 UTC m=+1310.146279530" watchObservedRunningTime="2026-10-30 22:22:59.303558517 +0000 UTC m=+1310.153071491"
Oct 30 22:22:59.347775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.347351    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:22:59.347775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.347406    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:22:59.657507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.657412    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"]
Oct 30 22:22:59.659249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.659218    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d0f9b0cc-326e-429d-9a32-1694cafc92a8" containerName="revoke-keys"
Oct 30 22:22:59.659408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.659392    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f9b0cc-326e-429d-9a32-1694cafc92a8" containerName="revoke-keys"
Oct 30 22:22:59.659723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.659705    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f9b0cc-326e-429d-9a32-1694cafc92a8" containerName="revoke-keys"
Oct 30 22:22:59.664582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.664557    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.694986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.694947    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.695173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.695072    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.695273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.695209    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pbv\" (UniqueName: \"kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.711547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.711512    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"]
Oct 30 22:22:59.796213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.796151    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.796385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.796227    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pbv\" (UniqueName: \"kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.796425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.796388    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.797425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.797392    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.799121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.799093    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.857144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.857079    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pbv\" (UniqueName: \"kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv\") pod \"maas-api-revoke-sub-ed8cc049-nwtl8\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:22:59.989828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:22:59.989794    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:23:00.195528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.195492    5028 generic.go:356] "Generic (PLEG): container finished" podID="11bc7ef2-5e28-4e1a-89b7-d607394c017e" containerID="46f42031ea491b78e9f53d48983691a44b1af29fec04a5ab59a6cd5b8ea6283c" exitCode=0
Oct 30 22:23:00.195726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.195567    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz" event={"ID":"11bc7ef2-5e28-4e1a-89b7-d607394c017e","Type":"ContainerDied","Data":"46f42031ea491b78e9f53d48983691a44b1af29fec04a5ab59a6cd5b8ea6283c"}
Oct 30 22:23:00.332859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.332806    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:23:00.347427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.347378    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:23:00.417571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.414920    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:23:00.421434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.421393    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"]
Oct 30 22:23:00.421595 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:00.421563    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe55cdf_9352_492f_8305_e60795b5c811.slice/crio-0a2e1fe36c4499aefbe56e6da37dadca6e5361e58114c08089121b59b60e6bef WatchSource:0}: Error finding container 0a2e1fe36c4499aefbe56e6da37dadca6e5361e58114c08089121b59b60e6bef: Status 404 returned error can't find the container with id 0a2e1fe36c4499aefbe56e6da37dadca6e5361e58114c08089121b59b60e6bef
Oct 30 22:23:00.512726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.512606    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca\") pod \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") "
Oct 30 22:23:00.512902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.512871    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca\") pod \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") "
Oct 30 22:23:00.513014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.512989    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca" pod "3b2f8add-e6c4-41c2-aa95-fae5695b1e61" (UID: "3b2f8add-e6c4-41c2-aa95-fae5695b1e61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:00.513118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.513097    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9\" (UniqueName: \"kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9\") pod \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\" (UID: \"3b2f8add-e6c4-41c2-aa95-fae5695b1e61\") "
Oct 30 22:23:00.513788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.513749    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:00.515115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.515087    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca" pod "3b2f8add-e6c4-41c2-aa95-fae5695b1e61" (UID: "3b2f8add-e6c4-41c2-aa95-fae5695b1e61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:00.515676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.515650    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9" pod "3b2f8add-e6c4-41c2-aa95-fae5695b1e61" (UID: "3b2f8add-e6c4-41c2-aa95-fae5695b1e61"). InnerVolumeSpecName "kube-api-access-6vxw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:00.615733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.615667    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:00.615733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:00.615723    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxw9\" (UniqueName: \"kubernetes.io/projected/3b2f8add-e6c4-41c2-aa95-fae5695b1e61-kube-api-access-6vxw9\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:01.211146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.211119    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"
Oct 30 22:23:01.211598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.211172    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl" event={"ID":"3b2f8add-e6c4-41c2-aa95-fae5695b1e61","Type":"ContainerDied","Data":"44c2a1f7e9f71642937035b2b96e148f401a61ca4d0fdeb7b090d8179d233a02"}
Oct 30 22:23:01.211598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.211236    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c2a1f7e9f71642937035b2b96e148f401a61ca4d0fdeb7b090d8179d233a02"
Oct 30 22:23:01.213597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.213562    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8" event={"ID":"2fe55cdf-9352-492f-8305-e60795b5c811","Type":"ContainerStarted","Data":"3290d1c9872132671c662d660fd04f18e10935a174194fac6611528ddf16a840"}
Oct 30 22:23:01.214079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.214057    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8" event={"ID":"2fe55cdf-9352-492f-8305-e60795b5c811","Type":"ContainerStarted","Data":"0a2e1fe36c4499aefbe56e6da37dadca6e5361e58114c08089121b59b60e6bef"}
Oct 30 22:23:01.332860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.328520    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8" podStartSLOduration=2.328499052 podStartE2EDuration="2.328499052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:01.326895117 +0000 UTC m=+1312.176408120" watchObservedRunningTime="2026-10-30 22:23:01.328499052 +0000 UTC m=+1312.178012026"
Oct 30 22:23:01.444817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.444768    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:23:01.445507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.445482    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:23:01.446505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.446306    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:23:01.494798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.494776    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:23:01.537933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.537904    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca\") pod \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") "
Oct 30 22:23:01.538107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.537940    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca\") pod \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") "
Oct 30 22:23:01.538107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.538055    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr\" (UniqueName: \"kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr\") pod \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\" (UID: \"11bc7ef2-5e28-4e1a-89b7-d607394c017e\") "
Oct 30 22:23:01.538404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.538381    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca" pod "11bc7ef2-5e28-4e1a-89b7-d607394c017e" (UID: "11bc7ef2-5e28-4e1a-89b7-d607394c017e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:01.539042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.539009    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:01.540251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.540229    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr" pod "11bc7ef2-5e28-4e1a-89b7-d607394c017e" (UID: "11bc7ef2-5e28-4e1a-89b7-d607394c017e"). InnerVolumeSpecName "kube-api-access-xrmxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:01.540329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.540288    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca" pod "11bc7ef2-5e28-4e1a-89b7-d607394c017e" (UID: "11bc7ef2-5e28-4e1a-89b7-d607394c017e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:01.640455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.640415    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11bc7ef2-5e28-4e1a-89b7-d607394c017e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:01.640455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:01.640447    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmxr\" (UniqueName: \"kubernetes.io/projected/11bc7ef2-5e28-4e1a-89b7-d607394c017e-kube-api-access-xrmxr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.224650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.224614    5028 generic.go:356] "Generic (PLEG): container finished" podID="2fe55cdf-9352-492f-8305-e60795b5c811" containerID="3290d1c9872132671c662d660fd04f18e10935a174194fac6611528ddf16a840" exitCode=0
Oct 30 22:23:02.225152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.224716    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8" event={"ID":"2fe55cdf-9352-492f-8305-e60795b5c811","Type":"ContainerDied","Data":"3290d1c9872132671c662d660fd04f18e10935a174194fac6611528ddf16a840"}
Oct 30 22:23:02.226314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.226293    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"
Oct 30 22:23:02.226422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.226321    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz" event={"ID":"11bc7ef2-5e28-4e1a-89b7-d607394c017e","Type":"ContainerDied","Data":"6dd2b9cf8df78c8d87a8b0f5b135bbc4bfd04d81e21cde9178ab7f3971b1c7a4"}
Oct 30 22:23:02.226422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.226357    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd2b9cf8df78c8d87a8b0f5b135bbc4bfd04d81e21cde9178ab7f3971b1c7a4"
Oct 30 22:23:02.453232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.453196    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.475641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.475563    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"]
Oct 30 22:23:02.476352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.476331    5028 cpu_manager.go:414] "containerMap: removing container" podUID="3b2f8add-e6c4-41c2-aa95-fae5695b1e61" containerName="revoke-keys"
Oct 30 22:23:02.476463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.476352    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2f8add-e6c4-41c2-aa95-fae5695b1e61" containerName="revoke-keys"
Oct 30 22:23:02.476463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.476417    5028 cpu_manager.go:414] "containerMap: removing container" podUID="11bc7ef2-5e28-4e1a-89b7-d607394c017e" containerName="revoke-keys"
Oct 30 22:23:02.476463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.476426    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11bc7ef2-5e28-4e1a-89b7-d607394c017e" containerName="revoke-keys"
Oct 30 22:23:02.476628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.476614    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="11bc7ef2-5e28-4e1a-89b7-d607394c017e" containerName="revoke-keys"
Oct 30 22:23:02.476679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.476640    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2f8add-e6c4-41c2-aa95-fae5695b1e61" containerName="revoke-keys"
Oct 30 22:23:02.480627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.480605    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.543751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.543718    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"]
Oct 30 22:23:02.556786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.555043    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.556786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.555201    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5n5\" (UniqueName: \"kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.556786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.555463    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.656968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.656931    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.657137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.657082    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.657200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.657187    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5n5\" (UniqueName: \"kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.657721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.657674    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.659530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.659496    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.717186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.717158    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5n5\" (UniqueName: \"kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5\") pod \"maas-api-revoke-sub-a2c67834-fhsdp\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:02.805421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:02.805336    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:03.180996 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:03.180963    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccc4a53_4ebc_41be_990e_2e279f496a32.slice/crio-1bd42a54a2eecc87f796f99c216c09b322859d00a72dee989d55800345fd9672 WatchSource:0}: Error finding container 1bd42a54a2eecc87f796f99c216c09b322859d00a72dee989d55800345fd9672: Status 404 returned error can't find the container with id 1bd42a54a2eecc87f796f99c216c09b322859d00a72dee989d55800345fd9672
Oct 30 22:23:03.183731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.183667    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"]
Oct 30 22:23:03.237253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.237215    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp" event={"ID":"5ccc4a53-4ebc-41be-990e-2e279f496a32","Type":"ContainerStarted","Data":"1bd42a54a2eecc87f796f99c216c09b322859d00a72dee989d55800345fd9672"}
Oct 30 22:23:03.418088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.418062    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:23:03.467973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.467934    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca\") pod \"2fe55cdf-9352-492f-8305-e60795b5c811\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") "
Oct 30 22:23:03.468142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.468018    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv\" (UniqueName: \"kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv\") pod \"2fe55cdf-9352-492f-8305-e60795b5c811\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") "
Oct 30 22:23:03.468142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.468060    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca\") pod \"2fe55cdf-9352-492f-8305-e60795b5c811\" (UID: \"2fe55cdf-9352-492f-8305-e60795b5c811\") "
Oct 30 22:23:03.468347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.468318    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca" pod "2fe55cdf-9352-492f-8305-e60795b5c811" (UID: "2fe55cdf-9352-492f-8305-e60795b5c811"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:03.468637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.468619    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:03.470140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.470115    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca" pod "2fe55cdf-9352-492f-8305-e60795b5c811" (UID: "2fe55cdf-9352-492f-8305-e60795b5c811"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:03.470245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.470223    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv" pod "2fe55cdf-9352-492f-8305-e60795b5c811" (UID: "2fe55cdf-9352-492f-8305-e60795b5c811"). InnerVolumeSpecName "kube-api-access-m7pbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:03.506313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.506284    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7126-0f93f2"]
Oct 30 22:23:03.507048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.507028    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2fe55cdf-9352-492f-8305-e60795b5c811" containerName="revoke-keys"
Oct 30 22:23:03.507135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.507048    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe55cdf-9352-492f-8305-e60795b5c811" containerName="revoke-keys"
Oct 30 22:23:03.507236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.507223    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe55cdf-9352-492f-8305-e60795b5c811" containerName="revoke-keys"
Oct 30 22:23:03.511373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.511348    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:03.561649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.561613    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7126-0f93f2"]
Oct 30 22:23:03.561847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.561700    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9v4z\""
Oct 30 22:23:03.570748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.570710    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wbd\" (UniqueName: \"kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd\") pod \"test-curl-7126-0f93f2\" (UID: \"e1d27d83-7455-452f-8b78-de7eeced0a45\") " pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:03.570958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.570944    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pbv\" (UniqueName: \"kubernetes.io/projected/2fe55cdf-9352-492f-8305-e60795b5c811-kube-api-access-m7pbv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:03.570999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.570965    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe55cdf-9352-492f-8305-e60795b5c811-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:03.672345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.672312    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wbd\" (UniqueName: \"kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd\") pod \"test-curl-7126-0f93f2\" (UID: \"e1d27d83-7455-452f-8b78-de7eeced0a45\") " pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:03.733295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.733258    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wbd\" (UniqueName: \"kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd\") pod \"test-curl-7126-0f93f2\" (UID: \"e1d27d83-7455-452f-8b78-de7eeced0a45\") " pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:03.825607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:03.825572    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:04.200126 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:04.200096    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1d27d83_7455_452f_8b78_de7eeced0a45.slice/crio-2908114c74f306436756812cb5252ebafdf295c9ac3237bf34a592921b52298a WatchSource:0}: Error finding container 2908114c74f306436756812cb5252ebafdf295c9ac3237bf34a592921b52298a: Status 404 returned error can't find the container with id 2908114c74f306436756812cb5252ebafdf295c9ac3237bf34a592921b52298a
Oct 30 22:23:04.202233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.202211    5028 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:23:04.202349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.202329    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7126-0f93f2"]
Oct 30 22:23:04.248518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.248488    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"
Oct 30 22:23:04.248905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.248521    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8" event={"ID":"2fe55cdf-9352-492f-8305-e60795b5c811","Type":"ContainerDied","Data":"0a2e1fe36c4499aefbe56e6da37dadca6e5361e58114c08089121b59b60e6bef"}
Oct 30 22:23:04.248905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.248580    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2e1fe36c4499aefbe56e6da37dadca6e5361e58114c08089121b59b60e6bef"
Oct 30 22:23:04.250471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.250445    5028 generic.go:356] "Generic (PLEG): container finished" podID="5ccc4a53-4ebc-41be-990e-2e279f496a32" containerID="c86abe0680fd452ac63e1be90a4152d5533102ff791acfbc512bc228179cb354" exitCode=0
Oct 30 22:23:04.250596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.250538    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp" event={"ID":"5ccc4a53-4ebc-41be-990e-2e279f496a32","Type":"ContainerDied","Data":"c86abe0680fd452ac63e1be90a4152d5533102ff791acfbc512bc228179cb354"}
Oct 30 22:23:04.252040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:04.252012    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7126-0f93f2" event={"ID":"e1d27d83-7455-452f-8b78-de7eeced0a45","Type":"ContainerStarted","Data":"2908114c74f306436756812cb5252ebafdf295c9ac3237bf34a592921b52298a"}
Oct 30 22:23:05.448921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.448895    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:05.502572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.502543    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5\" (UniqueName: \"kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5\") pod \"5ccc4a53-4ebc-41be-990e-2e279f496a32\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") "
Oct 30 22:23:05.502749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.502731    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca\") pod \"5ccc4a53-4ebc-41be-990e-2e279f496a32\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") "
Oct 30 22:23:05.502811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.502797    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca\") pod \"5ccc4a53-4ebc-41be-990e-2e279f496a32\" (UID: \"5ccc4a53-4ebc-41be-990e-2e279f496a32\") "
Oct 30 22:23:05.503193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.503161    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca" pod "5ccc4a53-4ebc-41be-990e-2e279f496a32" (UID: "5ccc4a53-4ebc-41be-990e-2e279f496a32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:05.503388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.503371    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:05.504772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.504746    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca" pod "5ccc4a53-4ebc-41be-990e-2e279f496a32" (UID: "5ccc4a53-4ebc-41be-990e-2e279f496a32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:05.504772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.504758    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5" pod "5ccc4a53-4ebc-41be-990e-2e279f496a32" (UID: "5ccc4a53-4ebc-41be-990e-2e279f496a32"). InnerVolumeSpecName "kube-api-access-gt5n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:05.605288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.605221    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5n5\" (UniqueName: \"kubernetes.io/projected/5ccc4a53-4ebc-41be-990e-2e279f496a32-kube-api-access-gt5n5\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:05.605288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.605262    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccc4a53-4ebc-41be-990e-2e279f496a32-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:05.762328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.762293    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"]
Oct 30 22:23:05.763083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.763063    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5ccc4a53-4ebc-41be-990e-2e279f496a32" containerName="revoke-keys"
Oct 30 22:23:05.763163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.763085    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccc4a53-4ebc-41be-990e-2e279f496a32" containerName="revoke-keys"
Oct 30 22:23:05.763284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.763270    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccc4a53-4ebc-41be-990e-2e279f496a32" containerName="revoke-keys"
Oct 30 22:23:05.767233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.767176    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:05.823735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.822244    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"]
Oct 30 22:23:05.910924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.910877    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:05.911079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.910965    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g6g\" (UniqueName: \"kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:05.911079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:05.911054    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.012505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.012467    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.012726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.012556    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g6g\" (UniqueName: \"kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.012726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.012641    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.013345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.013317    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.015467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.015445    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.053564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.053465    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"]
Oct 30 22:23:06.060148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.060116    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f57d2e-48ht5"]
Oct 30 22:23:06.071921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.071896    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g6g\" (UniqueName: \"kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g\") pod \"maas-api-revoke-sub-398dc751-6grwc\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.091237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.091201    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:06.281629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.281484    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp" event={"ID":"5ccc4a53-4ebc-41be-990e-2e279f496a32","Type":"ContainerDied","Data":"1bd42a54a2eecc87f796f99c216c09b322859d00a72dee989d55800345fd9672"}
Oct 30 22:23:06.283729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.282024    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd42a54a2eecc87f796f99c216c09b322859d00a72dee989d55800345fd9672"
Oct 30 22:23:06.283729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.281955    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"
Oct 30 22:23:06.506869 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:06.504857    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e06d734_1ede_46fe_82a0_4f6b7820f9b6.slice/crio-c0afd711e59f0c94c12ce458b95bee0c10f797ba8737a1093789f7dcc97f7d6f WatchSource:0}: Error finding container c0afd711e59f0c94c12ce458b95bee0c10f797ba8737a1093789f7dcc97f7d6f: Status 404 returned error can't find the container with id c0afd711e59f0c94c12ce458b95bee0c10f797ba8737a1093789f7dcc97f7d6f
Oct 30 22:23:06.506869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.504893    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"]
Oct 30 22:23:06.786887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:06.786791    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.015641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:07.015600    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.293602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:07.293565    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276c00e4-36c5-4810-a610-fc842cdab9bd" path="/var/lib/kubelet/pods/276c00e4-36c5-4810-a610-fc842cdab9bd/volumes"
Oct 30 22:23:07.295174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:07.295138    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc" event={"ID":"4e06d734-1ede-46fe-82a0-4f6b7820f9b6","Type":"ContainerStarted","Data":"d67e20d20630101fdb9ef25b31bfe93d79b64dcacded3dc46ac1be4522f3db57"}
Oct 30 22:23:07.295326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:07.295199    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc" event={"ID":"4e06d734-1ede-46fe-82a0-4f6b7820f9b6","Type":"ContainerStarted","Data":"c0afd711e59f0c94c12ce458b95bee0c10f797ba8737a1093789f7dcc97f7d6f"}
Oct 30 22:23:07.411534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:07.411491    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc" podStartSLOduration=2.411475144 podStartE2EDuration="2.411475144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.40621205 +0000 UTC m=+1318.255725024" watchObservedRunningTime="2026-10-30 22:23:07.411475144 +0000 UTC m=+1318.260988117"
Oct 30 22:23:08.305736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:08.305679    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7126-0f93f2" event={"ID":"e1d27d83-7455-452f-8b78-de7eeced0a45","Type":"ContainerStarted","Data":"12990d08170be852d53d0064d884c5bd5213b19d698a6c37fde88b91c3177d0b"}
Oct 30 22:23:08.307523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:08.307490    5028 generic.go:356] "Generic (PLEG): container finished" podID="4e06d734-1ede-46fe-82a0-4f6b7820f9b6" containerID="d67e20d20630101fdb9ef25b31bfe93d79b64dcacded3dc46ac1be4522f3db57" exitCode=0
Oct 30 22:23:08.307633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:08.307568    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc" event={"ID":"4e06d734-1ede-46fe-82a0-4f6b7820f9b6","Type":"ContainerDied","Data":"d67e20d20630101fdb9ef25b31bfe93d79b64dcacded3dc46ac1be4522f3db57"}
Oct 30 22:23:08.418036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:08.417981    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7126-0f93f2" podStartSLOduration=2.212186228 podStartE2EDuration="5.417967722s" totalImagesPullingTime="3.205781494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:08.415655128 +0000 UTC m=+1319.265168100" watchObservedRunningTime="2026-10-30 22:23:08.417967722 +0000 UTC m=+1319.267480694"
Oct 30 22:23:09.509923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.509894    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:09.680270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.680175    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca\") pod \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") "
Oct 30 22:23:09.680438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.680336    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca\") pod \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") "
Oct 30 22:23:09.680531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.680508    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g\" (UniqueName: \"kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g\") pod \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\" (UID: \"4e06d734-1ede-46fe-82a0-4f6b7820f9b6\") "
Oct 30 22:23:09.680836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.680807    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca" pod "4e06d734-1ede-46fe-82a0-4f6b7820f9b6" (UID: "4e06d734-1ede-46fe-82a0-4f6b7820f9b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:09.681251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.681227    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.682866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.682834    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca" pod "4e06d734-1ede-46fe-82a0-4f6b7820f9b6" (UID: "4e06d734-1ede-46fe-82a0-4f6b7820f9b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:09.683194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.683173    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g" pod "4e06d734-1ede-46fe-82a0-4f6b7820f9b6" (UID: "4e06d734-1ede-46fe-82a0-4f6b7820f9b6"). InnerVolumeSpecName "kube-api-access-t5g6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:09.783178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.783145    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5g6g\" (UniqueName: \"kubernetes.io/projected/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-kube-api-access-t5g6g\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.783178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.783176    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06d734-1ede-46fe-82a0-4f6b7820f9b6-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.858445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.858355    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7126-0f93f2"]
Oct 30 22:23:09.864650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:09.864619    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7126-0f93f2"]
Oct 30 22:23:10.029169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.029136    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7126-dce4f0"]
Oct 30 22:23:10.029944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.029921    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4e06d734-1ede-46fe-82a0-4f6b7820f9b6" containerName="revoke-keys"
Oct 30 22:23:10.029944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.029943    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e06d734-1ede-46fe-82a0-4f6b7820f9b6" containerName="revoke-keys"
Oct 30 22:23:10.030117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.029959    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" containerName="test-curl-7126-0f93f2"
Oct 30 22:23:10.030117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.029964    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" containerName="test-curl-7126-0f93f2"
Oct 30 22:23:10.030117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.030108    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e06d734-1ede-46fe-82a0-4f6b7820f9b6" containerName="revoke-keys"
Oct 30 22:23:10.030117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.030116    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" containerName="test-curl-7126-0f93f2"
Oct 30 22:23:10.043877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.043848    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:10.087169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.087134    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7126-dce4f0"]
Oct 30 22:23:10.189498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.189457    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt4n\" (UniqueName: \"kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n\") pod \"test-curl-7126-dce4f0\" (UID: \"97be1d00-3d31-4017-978f-c305a551685a\") " pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:10.290925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.290831    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt4n\" (UniqueName: \"kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n\") pod \"test-curl-7126-dce4f0\" (UID: \"97be1d00-3d31-4017-978f-c305a551685a\") " pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:10.328015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.327981    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"
Oct 30 22:23:10.328190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.328011    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc" event={"ID":"4e06d734-1ede-46fe-82a0-4f6b7820f9b6","Type":"ContainerDied","Data":"c0afd711e59f0c94c12ce458b95bee0c10f797ba8737a1093789f7dcc97f7d6f"}
Oct 30 22:23:10.328190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.328065    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0afd711e59f0c94c12ce458b95bee0c10f797ba8737a1093789f7dcc97f7d6f"
Oct 30 22:23:10.328331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.328311    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7126-0f93f2" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" containerName="test-curl-7126-0f93f2" containerID="cri-o://12990d08170be852d53d0064d884c5bd5213b19d698a6c37fde88b91c3177d0b" gracePeriod=2
Oct 30 22:23:10.349881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.349855    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt4n\" (UniqueName: \"kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n\") pod \"test-curl-7126-dce4f0\" (UID: \"97be1d00-3d31-4017-978f-c305a551685a\") " pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:10.356763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.356741    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:10.380139 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:10.380105    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7126-0f93f2\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:10.396991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.396951    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"]
Oct 30 22:23:10.402712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.402676    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.432627 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:10.431802    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7126-0f93f2\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:10.458426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.456064    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:23:10.458426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.456197    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:10.458426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.456262    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:10.458426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.456459    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:10.467224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.467193    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"]
Oct 30 22:23:10.504461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.503642    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.531208 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:10.531169    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7126-0f93f2\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:10.603372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.603277    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.603554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.603404    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wps\" (UniqueName: \"kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.603554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.603534    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.705006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.704968    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.705208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.705025    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wps\" (UniqueName: \"kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.705208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.705086    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.705656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.705630    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.707540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.707514    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:10.737792 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:10.737751    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97be1d00_3d31_4017_978f_c305a551685a.slice/crio-569e8d344dd63dda186a281bfef81a2e531419ef231bdc84e26ca8dd1fe40d38 WatchSource:0}: Error finding container 569e8d344dd63dda186a281bfef81a2e531419ef231bdc84e26ca8dd1fe40d38: Status 404 returned error can't find the container with id 569e8d344dd63dda186a281bfef81a2e531419ef231bdc84e26ca8dd1fe40d38
Oct 30 22:23:10.739322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.739303    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7126-dce4f0"]
Oct 30 22:23:10.764534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:10.764510    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wps\" (UniqueName: \"kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps\") pod \"maas-api-revoke-sub-4b8926cb-2jm6b\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:11.024079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.024037    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:11.032485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.032452    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"]
Oct 30 22:23:11.038423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.038395    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.064334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.064303    5028 scope.go:122] "RemoveContainer" containerID="a4132f192d7fd1df708c1deb165807ddc527ca7d6b485c609c7eccfa329bab68"
Oct 30 22:23:11.098466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.089248    5028 scope.go:122] "RemoveContainer" containerID="86fd417221393119dac304100b41d305d3bc036584970a4343fb7f85bfd6dda4"
Oct 30 22:23:11.098466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.093185    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-05e3625f-openshift-default-dockercfg-8l4lr\""
Oct 30 22:23:11.101577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.101551    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"]
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121434    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121502    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121675    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121768    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7llc\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121820    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121940    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.121978    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.122032    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.122586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.122064    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.125327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.125307    5028 scope.go:122] "RemoveContainer" containerID="7ac774904203990bda895b17cdbf38734aa41a08929ec1932887f675faacda78"
Oct 30 22:23:11.131345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.131320    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.144755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.144730    5028 scope.go:122] "RemoveContainer" containerID="910e865216405b3416daf487ddcc4b1dbf0bf2ce2836572d98b56d5c6ca6bfbc"
Oct 30 22:23:11.160891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.160872    5028 scope.go:122] "RemoveContainer" containerID="af43de7382b5607dc3b650a95ba041639fa21734cd23a3b0f91e1a77cdc5f2b4"
Oct 30 22:23:11.213933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.195944    5028 scope.go:122] "RemoveContainer" containerID="452ad910545e650d07fdb9c1c95be48c196f11fb20ee0bf5ddc9a0a0fdc6fec7"
Oct 30 22:23:11.242076 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:11.241740    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7126-0f93f2\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275427    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275624    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275712    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7llc\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275759    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275879    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275917    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275963    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.275993    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.276036    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.276518    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.277153    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.277377    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.277921    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.289762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.278378    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.347086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.346731    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7llc\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.367066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.350187    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.407793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.382294    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.407793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.405662    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.410158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.409923    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.453480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.453442    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7126-dce4f0" event={"ID":"97be1d00-3d31-4017-978f-c305a551685a","Type":"ContainerStarted","Data":"db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59"}
Oct 30 22:23:11.453667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.453643    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7126-dce4f0" event={"ID":"97be1d00-3d31-4017-978f-c305a551685a","Type":"ContainerStarted","Data":"569e8d344dd63dda186a281bfef81a2e531419ef231bdc84e26ca8dd1fe40d38"}
Oct 30 22:23:11.476418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.475671    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"]
Oct 30 22:23:11.550697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.550653    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.594489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.594020    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7126-dce4f0" podStartSLOduration=2.593998229 podStartE2EDuration="2.593998229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:11.583932217 +0000 UTC m=+1322.433445191" watchObservedRunningTime="2026-10-30 22:23:11.593998229 +0000 UTC m=+1322.443511205"
Oct 30 22:23:11.673707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.662969    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:11.711712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:11.708286    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:12.129078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.128990    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:12.161373 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:12.161336    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818ea594_fa7b_4161_b921_2888845dee5b.slice/crio-833caa19177364889a09ce13589fb2fec969fdf3f04cf4ef286468dc5006b0b3 WatchSource:0}: Error finding container 833caa19177364889a09ce13589fb2fec969fdf3f04cf4ef286468dc5006b0b3: Status 404 returned error can't find the container with id 833caa19177364889a09ce13589fb2fec969fdf3f04cf4ef286468dc5006b0b3
Oct 30 22:23:12.165953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.165908    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:12.166075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.166050    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:12.166160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.166142    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:12.166551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.166525    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"]
Oct 30 22:23:12.510746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.499932    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b" event={"ID":"4d070047-c20a-445c-b2ed-2d8f138543d0","Type":"ContainerStarted","Data":"d23544b7cbbbb8cf1e5e927bf581bfd81226e5a512bf174833395774db0b3d8a"}
Oct 30 22:23:12.510746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.499999    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b" event={"ID":"4d070047-c20a-445c-b2ed-2d8f138543d0","Type":"ContainerStarted","Data":"81cf0ea9695a6b1a32fa1fc1c5b7f4860c8ce62ae6be84a1a763bf79ce86b3bd"}
Oct 30 22:23:12.510746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.503432    5028 generic.go:356] "Generic (PLEG): container finished" podID="e1d27d83-7455-452f-8b78-de7eeced0a45" containerID="12990d08170be852d53d0064d884c5bd5213b19d698a6c37fde88b91c3177d0b" exitCode=137
Oct 30 22:23:12.510746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.509811    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" event={"ID":"818ea594-fa7b-4161-b921-2888845dee5b","Type":"ContainerStarted","Data":"2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"}
Oct 30 22:23:12.510746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.509865    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" event={"ID":"818ea594-fa7b-4161-b921-2888845dee5b","Type":"ContainerStarted","Data":"833caa19177364889a09ce13589fb2fec969fdf3f04cf4ef286468dc5006b0b3"}
Oct 30 22:23:12.663434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.663398    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:12.665377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.665312    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b" podStartSLOduration=2.665298673 podStartE2EDuration="2.665298673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:12.65972647 +0000 UTC m=+1323.509239444" watchObservedRunningTime="2026-10-30 22:23:12.665298673 +0000 UTC m=+1323.514811696"
Oct 30 22:23:12.694054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.694022    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:12.764943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.764857    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd\" (UniqueName: \"kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd\") pod \"e1d27d83-7455-452f-8b78-de7eeced0a45\" (UID: \"e1d27d83-7455-452f-8b78-de7eeced0a45\") "
Oct 30 22:23:12.766983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.766953    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd" pod "e1d27d83-7455-452f-8b78-de7eeced0a45" (UID: "e1d27d83-7455-452f-8b78-de7eeced0a45"). InnerVolumeSpecName "kube-api-access-d5wbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:12.782724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.782661    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podStartSLOduration=2.782646121 podStartE2EDuration="2.782646121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:12.77701367 +0000 UTC m=+1323.626526663" watchObservedRunningTime="2026-10-30 22:23:12.782646121 +0000 UTC m=+1323.632159094"
Oct 30 22:23:12.867068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:12.867031    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wbd\" (UniqueName: \"kubernetes.io/projected/e1d27d83-7455-452f-8b78-de7eeced0a45-kube-api-access-d5wbd\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.167842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.167750    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7126-dce4f0"]
Oct 30 22:23:13.176498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.176449    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7126-dce4f0"]
Oct 30 22:23:13.292969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.292925    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d27d83-7455-452f-8b78-de7eeced0a45" path="/var/lib/kubelet/pods/e1d27d83-7455-452f-8b78-de7eeced0a45/volumes"
Oct 30 22:23:13.523316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.523284    5028 generic.go:356] "Generic (PLEG): container finished" podID="4d070047-c20a-445c-b2ed-2d8f138543d0" containerID="d23544b7cbbbb8cf1e5e927bf581bfd81226e5a512bf174833395774db0b3d8a" exitCode=0
Oct 30 22:23:13.523510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.523326    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b" event={"ID":"4d070047-c20a-445c-b2ed-2d8f138543d0","Type":"ContainerDied","Data":"d23544b7cbbbb8cf1e5e927bf581bfd81226e5a512bf174833395774db0b3d8a"}
Oct 30 22:23:13.525053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.525033    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-0f93f2"
Oct 30 22:23:13.525182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.525097    5028 scope.go:122] "RemoveContainer" containerID="12990d08170be852d53d0064d884c5bd5213b19d698a6c37fde88b91c3177d0b"
Oct 30 22:23:13.525453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.525426    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7126-dce4f0" podUID="97be1d00-3d31-4017-978f-c305a551685a" containerName="test-curl-7126-dce4f0" containerID="cri-o://db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59" gracePeriod=2
Oct 30 22:23:13.671428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:13.671388    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:14.055510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.055467    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:14.547093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.546544    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:14.547661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.547629    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:23:14.725576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.725521    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:14.788725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.784064    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"]
Oct 30 22:23:14.788725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.785149    5028 cpu_manager.go:414] "containerMap: removing container" podUID="97be1d00-3d31-4017-978f-c305a551685a" containerName="test-curl-7126-dce4f0"
Oct 30 22:23:14.788725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.785170    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97be1d00-3d31-4017-978f-c305a551685a" containerName="test-curl-7126-dce4f0"
Oct 30 22:23:14.788725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.785378    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="97be1d00-3d31-4017-978f-c305a551685a" containerName="test-curl-7126-dce4f0"
Oct 30 22:23:14.812778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.807915    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"]
Oct 30 22:23:14.812778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.808053    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:14.874458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.874430    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ae20a62d-openshift-default-dockercfg-wfnv2\""
Oct 30 22:23:14.888448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.888410    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:14.899605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.898441    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:14.954432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.954363    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps\" (UniqueName: \"kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps\") pod \"4d070047-c20a-445c-b2ed-2d8f138543d0\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") "
Oct 30 22:23:14.954698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.954668    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca\") pod \"4d070047-c20a-445c-b2ed-2d8f138543d0\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") "
Oct 30 22:23:14.954824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.954805    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca\") pod \"4d070047-c20a-445c-b2ed-2d8f138543d0\" (UID: \"4d070047-c20a-445c-b2ed-2d8f138543d0\") "
Oct 30 22:23:14.975472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.968256    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca" pod "4d070047-c20a-445c-b2ed-2d8f138543d0" (UID: "4d070047-c20a-445c-b2ed-2d8f138543d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:14.975652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.975502    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca" pod "4d070047-c20a-445c-b2ed-2d8f138543d0" (UID: "4d070047-c20a-445c-b2ed-2d8f138543d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:14.979727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.955288    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.976043    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps" pod "4d070047-c20a-445c-b2ed-2d8f138543d0" (UID: "4d070047-c20a-445c-b2ed-2d8f138543d0"). InnerVolumeSpecName "kube-api-access-86wps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:14.999939    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000094    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000170    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8z9\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000271    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000402    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000467    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000758    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.000892    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.001162    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86wps\" (UniqueName: \"kubernetes.io/projected/4d070047-c20a-445c-b2ed-2d8f138543d0-kube-api-access-86wps\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.001187    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.004652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.001202    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d070047-c20a-445c-b2ed-2d8f138543d0-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:15.047849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.047816    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:15.103220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103112    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.103220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103194    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.103460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103235    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.103711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103674    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.103812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103764    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.103883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103824    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.103883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.103854    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104002    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104057    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104093    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8z9\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104238    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104438    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104509    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.104753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.104719    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.106149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.106124    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.106444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.106409    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.167601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.167563    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.167819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.167802    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8z9\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9\") pod \"e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") " pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.227373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.223514    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:15.227373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.224716    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:15.559206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.559165    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b" event={"ID":"4d070047-c20a-445c-b2ed-2d8f138543d0","Type":"ContainerDied","Data":"81cf0ea9695a6b1a32fa1fc1c5b7f4860c8ce62ae6be84a1a763bf79ce86b3bd"}
Oct 30 22:23:15.559365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.559232    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81cf0ea9695a6b1a32fa1fc1c5b7f4860c8ce62ae6be84a1a763bf79ce86b3bd"
Oct 30 22:23:15.559365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.559186    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"
Oct 30 22:23:15.760035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.760007    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:15.833199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.833160    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"]
Oct 30 22:23:15.835158 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:15.835129    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a8e4e9_b3dd_4338_8f65_160235583490.slice/crio-62d91cdcb1dcead7f46a8f34b0479efeacf2a3e494aa6abe41bc35b65935b81a WatchSource:0}: Error finding container 62d91cdcb1dcead7f46a8f34b0479efeacf2a3e494aa6abe41bc35b65935b81a: Status 404 returned error can't find the container with id 62d91cdcb1dcead7f46a8f34b0479efeacf2a3e494aa6abe41bc35b65935b81a
Oct 30 22:23:15.838231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.838199    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:15.838332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.838272    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:15.838332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.838301    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:15.842119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.842094    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n\" (UniqueName: \"kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n\") pod \"97be1d00-3d31-4017-978f-c305a551685a\" (UID: \"97be1d00-3d31-4017-978f-c305a551685a\") "
Oct 30 22:23:15.844162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.844138    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n" pod "97be1d00-3d31-4017-978f-c305a551685a" (UID: "97be1d00-3d31-4017-978f-c305a551685a"). InnerVolumeSpecName "kube-api-access-smt4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:15.944183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:15.944107    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smt4n\" (UniqueName: \"kubernetes.io/projected/97be1d00-3d31-4017-978f-c305a551685a-kube-api-access-smt4n\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.571766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.571719    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" event={"ID":"36a8e4e9-b3dd-4338-8f65-160235583490","Type":"ContainerStarted","Data":"b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"}
Oct 30 22:23:16.571970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.571790    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" event={"ID":"36a8e4e9-b3dd-4338-8f65-160235583490","Type":"ContainerStarted","Data":"62d91cdcb1dcead7f46a8f34b0479efeacf2a3e494aa6abe41bc35b65935b81a"}
Oct 30 22:23:16.573313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.573287    5028 generic.go:356] "Generic (PLEG): container finished" podID="97be1d00-3d31-4017-978f-c305a551685a" containerID="db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59" exitCode=137
Oct 30 22:23:16.573450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.573327    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:16.573450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.573389    5028 scope.go:122] "RemoveContainer" containerID="db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59"
Oct 30 22:23:16.585073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.585055    5028 scope.go:122] "RemoveContainer" containerID="db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59"
Oct 30 22:23:16.585352 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:16.585329    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59\": container with ID starting with db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59 not found: ID does not exist" containerID="db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59"
Oct 30 22:23:16.585420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.585359    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59"} err="failed to get container status \"db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59\": rpc error: code = NotFound desc = could not find container \"db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59\": container with ID starting with db26f6b9a485ab615771ad59c307ec13d923e09ed2c7764bb8b210b92560db59 not found: ID does not exist"
Oct 30 22:23:16.696233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.696180    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podStartSLOduration=2.696165337 podStartE2EDuration="2.696165337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:16.692094275 +0000 UTC m=+1327.541607250" watchObservedRunningTime="2026-10-30 22:23:16.696165337 +0000 UTC m=+1327.545678309"
Oct 30 22:23:16.740950 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:16.740906    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7126-dce4f0\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="97be1d00-3d31-4017-978f-c305a551685a" pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:16.749519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:16.749489    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.229737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:17.226204    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:17.232551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:17.232526    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:17.281368 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:17.281321    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7126-dce4f0\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="97be1d00-3d31-4017-978f-c305a551685a" pod="opendatahub/test-curl-7126-dce4f0"
Oct 30 22:23:17.296092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:17.296040    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97be1d00-3d31-4017-978f-c305a551685a" path="/var/lib/kubelet/pods/97be1d00-3d31-4017-978f-c305a551685a/volumes"
Oct 30 22:23:17.586531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:17.586448    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:17.587665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:17.587642    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:23:18.870331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:18.870287    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.031917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.031880    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.198601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.198476    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.430083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.430047    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.545946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.545910    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"]
Oct 30 22:23:19.546994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.546973    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4d070047-c20a-445c-b2ed-2d8f138543d0" containerName="revoke-keys"
Oct 30 22:23:19.547047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.546997    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d070047-c20a-445c-b2ed-2d8f138543d0" containerName="revoke-keys"
Oct 30 22:23:19.547240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.547226    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d070047-c20a-445c-b2ed-2d8f138543d0" containerName="revoke-keys"
Oct 30 22:23:19.553291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.553270    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.611387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.611357    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"]
Oct 30 22:23:19.650642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.650604    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:19.650856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.650742    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:19.650856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.650745    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:23:19.654084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.654051    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:19.705908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.705871    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwth\" (UniqueName: \"kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.706078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.705994    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.706163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.706133    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.786489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.786458    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"]
Oct 30 22:23:19.793295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.793266    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.817954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.817363    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.824573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.824540    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.825085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.825041    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwth\" (UniqueName: \"kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.825619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.818316    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.848536    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.874157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.872325    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.895525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.895488    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"]
Oct 30 22:23:19.898972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.898910    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwth\" (UniqueName: \"kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth\") pod \"maas-api-revoke-sub-8f705abe-hb7rv\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927029    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927153    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927192    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927281    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnb4\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927339    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927369    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927428    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927456    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:19.936797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:19.927497    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.062768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.062532    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.069937    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070069    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070104    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070198    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnb4\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070260    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070292    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070354    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070382    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.070419    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.071292    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.072621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.071573    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.075055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.075024    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.075882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.075456    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.076318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.076293    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.078514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.078240    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.158721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.141158    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.158721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.150741    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"]
Oct 30 22:23:20.182454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.165655    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.182454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.167524    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnb4\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.182454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.170847    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:20.182454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.174034    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"]
Oct 30 22:23:20.235060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.234908    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.243806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.243770    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"]
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.269391    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.276844    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277047    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277079    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277177    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277273    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvnl\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277354    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277520    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277548    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.277591    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.278008    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"]
Oct 30 22:23:20.279145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.278112    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.419653    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.419745    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.420442    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430180    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430263    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430378    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430408    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430449    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59x2\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430511    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430642    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430779    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430887    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.430961    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431049    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431086    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431129    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431165    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431240    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvnl\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431303    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.431333    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.456086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.440306    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:20.476920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.464052    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.476920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.464316    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.476920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.465347    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.476920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.474488    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.476920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.476263    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.553036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.552943    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvnl\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.557076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.557029    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562159    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562248    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562378    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562405    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562437    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59x2\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562486    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562523    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562588    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.562644    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.563118    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.563916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.563637    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.576257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.574941    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.576257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.575217    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.576601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.576560    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.580467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.580394    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.581877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.581231    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.648102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.648045    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59x2\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.656285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.656261    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.696083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.696042    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"]
Oct 30 22:23:20.702644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.702614    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.760881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.760847    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"]
Oct 30 22:23:20.827667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.827635    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:20.870265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.870180    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.870641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.870301    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdl24\" (UniqueName: \"kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.870641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.870518    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.889226 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:20.889184    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fc6480_f647_45a4_a1da_8263220c3a5c.slice/crio-03a11d6a340362346d66d4dfd877847fed10fab65ae558ab617ac6da43451dd3 WatchSource:0}: Error finding container 03a11d6a340362346d66d4dfd877847fed10fab65ae558ab617ac6da43451dd3: Status 404 returned error can't find the container with id 03a11d6a340362346d66d4dfd877847fed10fab65ae558ab617ac6da43451dd3
Oct 30 22:23:20.897530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.897414    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"]
Oct 30 22:23:20.949404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.949145    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:20.972064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.972034    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.972199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.972184    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdl24\" (UniqueName: \"kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.972628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.972368    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.973277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.973253    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.975733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.975387    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:20.992062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.990498    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"]
Oct 30 22:23:20.992062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.990638    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:20.992062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.991188    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:20.992062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:20.991260    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.121220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.121143    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdl24\" (UniqueName: \"kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24\") pod \"maas-api-revoke-sub-3edaca53-nt87t\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:21.237036 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:21.237004    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a390ea_157d_4a39_be2f_a506f548304a.slice/crio-5429d00f64b654e7239816cef8d53a5545866f2b107161de21e5b632df769f94 WatchSource:0}: Error finding container 5429d00f64b654e7239816cef8d53a5545866f2b107161de21e5b632df769f94: Status 404 returned error can't find the container with id 5429d00f64b654e7239816cef8d53a5545866f2b107161de21e5b632df769f94
Oct 30 22:23:21.240102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.240079    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"]
Oct 30 22:23:21.240985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.240948    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.241053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.241035    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.241095 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.241083    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.328669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.328639    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:21.386804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.386719    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"]
Oct 30 22:23:21.393091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.393048    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.393219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.393139    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.393219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.393178    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:21.653592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.653543    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"]
Oct 30 22:23:21.662277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.662249    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.685636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.685604    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" event={"ID":"21a390ea-157d-4a39-be2f-a506f548304a","Type":"ContainerStarted","Data":"918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf"}
Oct 30 22:23:21.685796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.685654    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" event={"ID":"21a390ea-157d-4a39-be2f-a506f548304a","Type":"ContainerStarted","Data":"5429d00f64b654e7239816cef8d53a5545866f2b107161de21e5b632df769f94"}
Oct 30 22:23:21.689549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.689458    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv" event={"ID":"f7fc6480-f647-45a4-a1da-8263220c3a5c","Type":"ContainerStarted","Data":"b7f3763a4e9b79c20ad232711cd973f22eccb1c1668958c9d58d999ffc3b53b7"}
Oct 30 22:23:21.689549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.689514    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv" event={"ID":"f7fc6480-f647-45a4-a1da-8263220c3a5c","Type":"ContainerStarted","Data":"03a11d6a340362346d66d4dfd877847fed10fab65ae558ab617ac6da43451dd3"}
Oct 30 22:23:21.691456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.691428    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" event={"ID":"a2ac3c25-1344-4a8b-9499-8305d610254c","Type":"ContainerStarted","Data":"bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460"}
Oct 30 22:23:21.691565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.691476    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" event={"ID":"a2ac3c25-1344-4a8b-9499-8305d610254c","Type":"ContainerStarted","Data":"b6f1a255b056e228315aa95ae8f6519ca52961eaca80e78a85ebe464a3b7f089"}
Oct 30 22:23:21.692386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.692307    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.692386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.692358    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.692652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.692626    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.692764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.692672    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.692764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.692738    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxbq\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.692851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.692835    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.693455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.693430    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.694235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.694217    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" event={"ID":"5ec93019-94ab-409f-8d83-a9f449a03ef6","Type":"ContainerStarted","Data":"d0be13a468bec3d5782a4e5b943c01485ea4bbadba947a8c82952e8970550970"}
Oct 30 22:23:21.694290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.694252    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" event={"ID":"5ec93019-94ab-409f-8d83-a9f449a03ef6","Type":"ContainerStarted","Data":"09700dbda5f37677c94f2dbcc084382d5934435342c9fc50e73a551a016aa894"}
Oct 30 22:23:21.715814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.715777    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"]
Oct 30 22:23:21.721550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.720220    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7vbmp\""
Oct 30 22:23:21.721550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.720509    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:23:21.721550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.720587    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:23:21.721550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.720636    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:23:21.766974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.766942    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"]
Oct 30 22:23:21.796482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796438    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.796653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796492    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.796653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796532    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxbq\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.796653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796622    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.796851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796675    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.796851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796767    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.796951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.796853    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.802785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.802760    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.806331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.806285    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.807944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.807854    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.807944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.807863    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.808538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.808487    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.828154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.828120    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:21.861473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.861409    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.863618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.863567    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxbq\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qjq6b\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:21.917162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.915970    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podStartSLOduration=1.915953985 podStartE2EDuration="1.915953985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:21.913997233 +0000 UTC m=+1332.763510207" watchObservedRunningTime="2026-10-30 22:23:21.915953985 +0000 UTC m=+1332.765466958"
Oct 30 22:23:21.965754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.963792    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:21.988462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:21.984963    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:22.066950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.066649    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podStartSLOduration=2.066627701 podStartE2EDuration="2.066627701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.062405325 +0000 UTC m=+1332.911918324" watchObservedRunningTime="2026-10-30 22:23:22.066627701 +0000 UTC m=+1332.916140675"
Oct 30 22:23:22.191223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.189937    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podStartSLOduration=3.18991857 podStartE2EDuration="3.18991857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.182669302 +0000 UTC m=+1333.032182344" watchObservedRunningTime="2026-10-30 22:23:22.18991857 +0000 UTC m=+1333.039431546"
Oct 30 22:23:22.270646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.270418    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:22.559472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.547320    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv" podStartSLOduration=3.547281243 podStartE2EDuration="3.547281243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.370661631 +0000 UTC m=+1333.220174604" watchObservedRunningTime="2026-10-30 22:23:22.547281243 +0000 UTC m=+1333.396794217"
Oct 30 22:23:22.906187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.574989    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:22.906479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.906449    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:22.906894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.468885    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:22.907121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.899778    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:22.907253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.907234    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:22.980658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.980616    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:22.981301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:22.981264    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:23.190713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.164642    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"]
Oct 30 22:23:23.413258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.412440    5028 generic.go:356] "Generic (PLEG): container finished" podID="f7fc6480-f647-45a4-a1da-8263220c3a5c" containerID="b7f3763a4e9b79c20ad232711cd973f22eccb1c1668958c9d58d999ffc3b53b7" exitCode=0
Oct 30 22:23:23.450621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.450584    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:23.450920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.450890    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:23.535744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.522777    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:23.861087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.861048    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:23.921802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.900074    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:23.972789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.959112    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:23.972789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.959174    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:23.980907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:23.980736    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv" event={"ID":"f7fc6480-f647-45a4-a1da-8263220c3a5c","Type":"ContainerDied","Data":"b7f3763a4e9b79c20ad232711cd973f22eccb1c1668958c9d58d999ffc3b53b7"}
Oct 30 22:23:24.014434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.014343    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t" event={"ID":"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd","Type":"ContainerStarted","Data":"de7461a5600d03b52dab4330a25c9b2d150073996924717ccf0039dc4bc1b48f"}
Oct 30 22:23:24.110424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.110359    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t" podStartSLOduration=4.110336605 podStartE2EDuration="4.110336605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:24.011619273 +0000 UTC m=+1334.861132250" watchObservedRunningTime="2026-10-30 22:23:24.110336605 +0000 UTC m=+1334.959849583"
Oct 30 22:23:24.127918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.127870    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:24.208653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.208618    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.216610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.216178    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:24.380649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.380510    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.475728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.463449    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:24.475728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.463506    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:24.503718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.502769    5028 generic.go:356] "Generic (PLEG): container finished" podID="dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" containerID="452e3994f5a56d11f5c1a76c6a0c9e839af0193ac7b43f4f2d8eaa860d875cca" exitCode=0
Oct 30 22:23:24.503718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.503576    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t" event={"ID":"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd","Type":"ContainerDied","Data":"452e3994f5a56d11f5c1a76c6a0c9e839af0193ac7b43f4f2d8eaa860d875cca"}
Oct 30 22:23:24.539722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.536505    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" event={"ID":"27ed34d1-04a4-4d50-86ba-b9dbfce6246f","Type":"ContainerStarted","Data":"f0b4285017939b32988fead6b5fdd29c00b0fd9a6875007e137b155b0187df03"}
Oct 30 22:23:24.562715 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:24.543944    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd69d1f0_6d5f_49cd_9f3f_0c12cf2257dd.slice/crio-452e3994f5a56d11f5c1a76c6a0c9e839af0193ac7b43f4f2d8eaa860d875cca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd69d1f0_6d5f_49cd_9f3f_0c12cf2257dd.slice/crio-conmon-452e3994f5a56d11f5c1a76c6a0c9e839af0193ac7b43f4f2d8eaa860d875cca.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:23:24.645779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.631993    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.693713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.688791    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:24.846709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.846119    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:24.846709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.846176    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:24.875724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.858674    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.975492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.975456    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:24.975761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:24.975732    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:25.547710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.544840    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:25.547710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.544920    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:25.705791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.698857    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:25.906591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.853806    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:25.906591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.853868    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:25.984769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.971404    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:25.984769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:25.971462    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:26.024726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.012605    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:23:26.024726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.013141    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:23:26.101774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.095237    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.281980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.281945    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:26.282776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.282731    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.413304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.394987    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca\") pod \"f7fc6480-f647-45a4-a1da-8263220c3a5c\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") "
Oct 30 22:23:26.413304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.395069    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth\" (UniqueName: \"kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth\") pod \"f7fc6480-f647-45a4-a1da-8263220c3a5c\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") "
Oct 30 22:23:26.413304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.395225    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca\") pod \"f7fc6480-f647-45a4-a1da-8263220c3a5c\" (UID: \"f7fc6480-f647-45a4-a1da-8263220c3a5c\") "
Oct 30 22:23:26.414364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.414040    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca" pod "f7fc6480-f647-45a4-a1da-8263220c3a5c" (UID: "f7fc6480-f647-45a4-a1da-8263220c3a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:26.414995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.414958    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:26.463518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.463459    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca" pod "f7fc6480-f647-45a4-a1da-8263220c3a5c" (UID: "f7fc6480-f647-45a4-a1da-8263220c3a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:26.545860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.533417    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fc6480-f647-45a4-a1da-8263220c3a5c-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:26.545860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.544829    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:26.545860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.544887    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:26.545860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.545060    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth" pod "f7fc6480-f647-45a4-a1da-8263220c3a5c" (UID: "f7fc6480-f647-45a4-a1da-8263220c3a5c"). InnerVolumeSpecName "kube-api-access-6nwth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:26.608850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.607063    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.717765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.707868    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nwth\" (UniqueName: \"kubernetes.io/projected/f7fc6480-f647-45a4-a1da-8263220c3a5c-kube-api-access-6nwth\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:26.904712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.903811    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.950830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.908135    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:23:26.950830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.934669    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:26.950830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.934755    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:26.964587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.964548    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:27.016779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:26.999013    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:27.046428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.046370    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:23:27.105774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.101836    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.105774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.055781    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.105774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.102826    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:23:27.105774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.102911    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:23:27.325902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.325799    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.326508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.326483    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:23:27.428211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.428167    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:23:27.557721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.548982    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:23:27.716772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.582312    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:27.955719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.933833    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:27.956080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.629883    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:23:28.004852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.632127    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:28.005127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.889428    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:28.005254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.005228    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:28.005383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:27.907849    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:23:28.118958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.069973    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:28.118958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.070056    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:28.200736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.199250    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:28.238920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.238856    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:23:28.278252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.247052    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"
Oct 30 22:23:28.313709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.247343    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:28.326747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.322294    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded"
Oct 30 22:23:28.326943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.326761    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:28.487713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.480169    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.539989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.539888    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:28.830339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.830299    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:28.830951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.612272    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:23:28.856986 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:28.590936    5028 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.176s"
Oct 30 22:23:28.857373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.857351    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:23:28.857572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.857542    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv" event={"ID":"f7fc6480-f647-45a4-a1da-8263220c3a5c","Type":"ContainerDied","Data":"03a11d6a340362346d66d4dfd877847fed10fab65ae558ab617ac6da43451dd3"}
Oct 30 22:23:28.857757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.857729    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03a11d6a340362346d66d4dfd877847fed10fab65ae558ab617ac6da43451dd3"
Oct 30 22:23:28.857974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.857958    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:23:28.858308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.858292    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:23:28.858532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.858519    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:23:28.858730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.858717    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:23:28.859441 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.829994    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:28.859647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.859612    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:28.881063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.881004    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.954161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.954128    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:28.954472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:28.954426    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:29.031738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.027789    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:23:29.031738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.027728    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:29.111827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.108045    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:29.211933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.211875    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:29.245758 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:29.222815    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:23:29.246123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.246084    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:29.318831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.310204    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:29.380772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.338111    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:29.381119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.345109    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:23:29.381390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.345196    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:29.403407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.403362    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t" podUID="9b57c28a-0d5d-4eac-bee0-7f18f78aac56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:23:29.642802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.632825    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:29.642802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.632895    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:29.642802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.634158    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca\") pod \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") "
Oct 30 22:23:29.642802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.634206    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca\") pod \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") "
Oct 30 22:23:29.642802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.634346    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24\" (UniqueName: \"kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24\") pod \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\" (UID: \"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd\") "
Oct 30 22:23:29.643977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.643933    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t" podUID="9b57c28a-0d5d-4eac-bee0-7f18f78aac56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 22:23:29.865143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.851782    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:23:29.969581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:29.965356    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca" pod "dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" (UID: "dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:30.012082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.012035    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:30.012354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.012325    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:30.169719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.156376    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:30.169719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.157035    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:23:30.287771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.240299    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:23:30.287771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.257005    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:30.287771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.257572    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.325895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.325844    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:30.382832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.382783    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:23:30.424715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.410785    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:30.506768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.493258    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:30.506768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.493325    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:30.531757 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:30.507608    5028 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.214s"
Oct 30 22:23:30.545150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.514744    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"
Oct 30 22:23:30.569762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.514946    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:30.570336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.569980    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:30.570336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.566715    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:30.576822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.576796    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:23:30.602710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.588535    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t" event={"ID":"dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd","Type":"ContainerDied","Data":"de7461a5600d03b52dab4330a25c9b2d150073996924717ccf0039dc4bc1b48f"}
Oct 30 22:23:30.603013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.602987    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7461a5600d03b52dab4330a25c9b2d150073996924717ccf0039dc4bc1b48f"
Oct 30 22:23:30.603139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.593865    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca" pod "dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" (UID: "dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:30.749884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.678153    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.827369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.827279    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:30.874158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.869385    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:30.874158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.869449    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:30.874158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.869537    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:30.986712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.977247    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:30.986712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:30.977301    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:31.193774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.193336    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24" pod "dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" (UID: "dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd"). InnerVolumeSpecName "kube-api-access-wdl24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:31.194035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.193853    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.325719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.322173    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdl24\" (UniqueName: \"kubernetes.io/projected/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd-kube-api-access-wdl24\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:31.606712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.461635    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:23:31.607077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.607038    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 22:23:31.772771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.713411    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:23:31.772771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.732257    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:23:31.772771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.732629    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:23:31.772771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.765571    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:23:31.772771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.766046    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:23:31.798895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.798850    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 30 22:23:31.799178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.799152    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:23:31.844509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.844471    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 22:23:31.844802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.844770    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 22:23:31.930746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:31.912436    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:23:32.050789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.046980    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:32.050789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.047046    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:32.052184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.052147    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.366871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.366773    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:32.367109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.367081    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.716720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.715428    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.732817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.732770    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.733438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.733407    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.792745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.791081    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.792745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.791490    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:23:32.801834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.801793    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:32.954117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.954077    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:32.962825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.962778    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:32.963056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:32.956184    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:23:33.127144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.061672    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.128129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.128076    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded"
Oct 30 22:23:33.247835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.247784    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.378818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.364765    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.508710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.505774    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:33.508710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.505842    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.732713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.717748    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.822577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.733350    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.822577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.734579    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.901135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.890919    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"]
Oct 30 22:23:33.930720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.903731    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f7fc6480-f647-45a4-a1da-8263220c3a5c" containerName="revoke-keys"
Oct 30 22:23:33.939146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.939113    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fc6480-f647-45a4-a1da-8263220c3a5c" containerName="revoke-keys"
Oct 30 22:23:33.939475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.939459    5028 cpu_manager.go:414] "containerMap: removing container" podUID="dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" containerName="revoke-keys"
Oct 30 22:23:33.939572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.939560    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" containerName="revoke-keys"
Oct 30 22:23:33.940334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.940308    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" containerName="revoke-keys"
Oct 30 22:23:33.940520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.940505    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fc6480-f647-45a4-a1da-8263220c3a5c" containerName="revoke-keys"
Oct 30 22:23:33.965160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.954631    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:33.965160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.895091    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:23:33.965160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.895140    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:23:33.965901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.895053    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:23:33.987164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.987045    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:33.992530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.992477    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.002741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.998484    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:23:34.002741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:33.998536    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 30 22:23:34.102882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.102834    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.142211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.142153    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.196558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.196527    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:23:34.272620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.272530    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:34.301160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.301127    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:23:34.328043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.312107    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"]
Oct 30 22:23:34.441814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.440835    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.441814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.441164    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.441814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.441333    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwrx\" (UniqueName: \"kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.467770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.461071    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:23:34.537897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.537805    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:34.538188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.538152    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.576842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.576799    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:34.577137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.577104    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.597111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.597070    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.618007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.617963    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwrx\" (UniqueName: \"kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.646390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.600453    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.647366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.646658    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:34.700713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.697265    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.742908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.742859    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.743596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.743561    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.832655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.743944    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.965889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.965841    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.967347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.967316    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwrx\" (UniqueName: \"kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:35.199773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.970767    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.209967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:34.980779    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.210380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.009202    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d746664c-v8fnz\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:35.369589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.335976    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.370001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.369959    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.381793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.380844    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:35.666820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.666767    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:35.667114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.667073    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:35.865708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.831014    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:35.935196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.926778    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.935535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.935501    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:35.936192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.936040    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:35.936513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.936481    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.980677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.980598    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.989851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:35.989812    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.009261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.009226    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:36.018821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.018784    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.019279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.015266    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:23:36.054739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.031651    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:36.054739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.039314    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.054739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.039791    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:36.215793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.215660    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.249522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.249486    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:36.306708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.291382    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.314052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.314007    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:23:36.358286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.351814    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.368171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.358605    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:23:36.477090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.476741    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:23:36.631229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.631193    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:36.864726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.864604    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:36.865001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.864968    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.950853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.950826    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:23:36.966491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.966453    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.986720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.985874    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:23:36.990711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.987263    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:36.990711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:36.987321    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.025576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.025506    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:37.025796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.025625    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.040401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.039773    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.375175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.375019    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.429058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.429017    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.709949 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:37.686437    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d568fd_de6c_43d7_bfe5_6d7137646a54.slice/crio-f6fd9e9fe3f0bd6fe1a540dd62d4838fe69acfb540490943e3cd1e73ecd6b3e6 WatchSource:0}: Error finding container f6fd9e9fe3f0bd6fe1a540dd62d4838fe69acfb540490943e3cd1e73ecd6b3e6: Status 404 returned error can't find the container with id f6fd9e9fe3f0bd6fe1a540dd62d4838fe69acfb540490943e3cd1e73ecd6b3e6
Oct 30 22:23:37.884819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.876725    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:23:37.884819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.876857    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:37.884819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.876913    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:23:37.884819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.876933    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"]
Oct 30 22:23:37.919749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.919714    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:23:37.989109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.989077    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:37.989765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:37.989730    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.071782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.071718    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:38.072209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.072134    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.083593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.083559    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:38.087998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.087967    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:23:38.166090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.165906    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:38.617833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.617792    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"]
Oct 30 22:23:38.625052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.625025    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.661243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.661203    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"]
Oct 30 22:23:38.676083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.673758    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:38.683291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.683263    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:23:38.796826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.796738    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.796826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.796797    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv55f\" (UniqueName: \"kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.797042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.797020    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.894278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.891794    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz" event={"ID":"19d568fd-de6c-43d7-bfe5-6d7137646a54","Type":"ContainerStarted","Data":"5a807bc000e5e53249fca87bc4c811ddedd98374ecf8e2b3346345b974c1025f"}
Oct 30 22:23:38.894278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.891859    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz" event={"ID":"19d568fd-de6c-43d7-bfe5-6d7137646a54","Type":"ContainerStarted","Data":"f6fd9e9fe3f0bd6fe1a540dd62d4838fe69acfb540490943e3cd1e73ecd6b3e6"}
Oct 30 22:23:38.899422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.899387    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.899554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.899455    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv55f\" (UniqueName: \"kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.899616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.899600    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.903036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.901930    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.903036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.902666    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.962945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.962915    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv55f\" (UniqueName: \"kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f\") pod \"maas-api-revoke-sub-a0253345-m8ddq\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:38.966999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:38.966774    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:39.007142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:39.007083    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz" podStartSLOduration=8.007067765 podStartE2EDuration="8.007067765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:39.003303325 +0000 UTC m=+1349.852816309" watchObservedRunningTime="2026-10-30 22:23:39.007067765 +0000 UTC m=+1349.856580737"
Oct 30 22:23:39.366542 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:39.366507    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb725fd6a_7e97_45ab_baec_a5a5c35970cf.slice/crio-b6cfb83b71ff1d386748cb73081cb4fe538e16363b1ecb78aa59fe1db4c47011 WatchSource:0}: Error finding container b6cfb83b71ff1d386748cb73081cb4fe538e16363b1ecb78aa59fe1db4c47011: Status 404 returned error can't find the container with id b6cfb83b71ff1d386748cb73081cb4fe538e16363b1ecb78aa59fe1db4c47011
Oct 30 22:23:39.367189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:39.367162    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"]
Oct 30 22:23:39.902089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:39.901990    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq" event={"ID":"b725fd6a-7e97-45ab-baec-a5a5c35970cf","Type":"ContainerStarted","Data":"2169a396bda8ce4f8e2b136eb3a5845094a3be04ef251c585ad8eeb1515e5c24"}
Oct 30 22:23:39.902254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:39.902088    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq" event={"ID":"b725fd6a-7e97-45ab-baec-a5a5c35970cf","Type":"ContainerStarted","Data":"b6cfb83b71ff1d386748cb73081cb4fe538e16363b1ecb78aa59fe1db4c47011"}
Oct 30 22:23:39.905628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:39.905596    5028 generic.go:356] "Generic (PLEG): container finished" podID="19d568fd-de6c-43d7-bfe5-6d7137646a54" containerID="5a807bc000e5e53249fca87bc4c811ddedd98374ecf8e2b3346345b974c1025f" exitCode=0
Oct 30 22:23:39.905821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:39.905696    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz" event={"ID":"19d568fd-de6c-43d7-bfe5-6d7137646a54","Type":"ContainerDied","Data":"5a807bc000e5e53249fca87bc4c811ddedd98374ecf8e2b3346345b974c1025f"}
Oct 30 22:23:40.018381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:40.018311    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq" podStartSLOduration=2.01829106 podStartE2EDuration="2.01829106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:40.014617269 +0000 UTC m=+1350.864130244" watchObservedRunningTime="2026-10-30 22:23:40.01829106 +0000 UTC m=+1350.867804034"
Oct 30 22:23:40.917290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:40.916963    5028 generic.go:356] "Generic (PLEG): container finished" podID="b725fd6a-7e97-45ab-baec-a5a5c35970cf" containerID="2169a396bda8ce4f8e2b136eb3a5845094a3be04ef251c585ad8eeb1515e5c24" exitCode=0
Oct 30 22:23:40.917290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:40.917065    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq" event={"ID":"b725fd6a-7e97-45ab-baec-a5a5c35970cf","Type":"ContainerDied","Data":"2169a396bda8ce4f8e2b136eb3a5845094a3be04ef251c585ad8eeb1515e5c24"}
Oct 30 22:23:40.968939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:40.968903    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:40.969080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:40.968968    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:41.125763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.119283    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:41.167361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.167269    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx\" (UniqueName: \"kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx\") pod \"19d568fd-de6c-43d7-bfe5-6d7137646a54\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") "
Oct 30 22:23:41.167361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.167310    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca\") pod \"19d568fd-de6c-43d7-bfe5-6d7137646a54\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") "
Oct 30 22:23:41.167635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.167611    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca\") pod \"19d568fd-de6c-43d7-bfe5-6d7137646a54\" (UID: \"19d568fd-de6c-43d7-bfe5-6d7137646a54\") "
Oct 30 22:23:41.167825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.167802    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca" pod "19d568fd-de6c-43d7-bfe5-6d7137646a54" (UID: "19d568fd-de6c-43d7-bfe5-6d7137646a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:41.168337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.168302    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:41.170531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.170190    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx" pod "19d568fd-de6c-43d7-bfe5-6d7137646a54" (UID: "19d568fd-de6c-43d7-bfe5-6d7137646a54"). InnerVolumeSpecName "kube-api-access-8vwrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:41.170531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.170333    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca" pod "19d568fd-de6c-43d7-bfe5-6d7137646a54" (UID: "19d568fd-de6c-43d7-bfe5-6d7137646a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:41.270269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.270230    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d568fd-de6c-43d7-bfe5-6d7137646a54-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:41.270269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.270268    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwrx\" (UniqueName: \"kubernetes.io/projected/19d568fd-de6c-43d7-bfe5-6d7137646a54-kube-api-access-8vwrx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:41.566251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.566215    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"]
Oct 30 22:23:41.566994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.566978    5028 cpu_manager.go:414] "containerMap: removing container" podUID="19d568fd-de6c-43d7-bfe5-6d7137646a54" containerName="revoke-keys"
Oct 30 22:23:41.566994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.566994    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d568fd-de6c-43d7-bfe5-6d7137646a54" containerName="revoke-keys"
Oct 30 22:23:41.567155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.567146    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d568fd-de6c-43d7-bfe5-6d7137646a54" containerName="revoke-keys"
Oct 30 22:23:41.571356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.571334    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.647589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.624868    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"]
Oct 30 22:23:41.676906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.676865    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.677152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.677126    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.677214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.677181    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtr9\" (UniqueName: \"kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.778615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.778578    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.778801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.778626    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtr9\" (UniqueName: \"kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.778853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.778805    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.779325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.779304    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.781290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.781267    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.837857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.837784    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtr9\" (UniqueName: \"kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9\") pod \"maas-api-revoke-sub-6d1d18a5-cn92n\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.888460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.888425    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:41.932118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.932075    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" event={"ID":"27ed34d1-04a4-4d50-86ba-b9dbfce6246f","Type":"ContainerStarted","Data":"1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c"}
Oct 30 22:23:41.932334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.932180    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:41.934126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.934101    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"
Oct 30 22:23:41.934338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.934174    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz" event={"ID":"19d568fd-de6c-43d7-bfe5-6d7137646a54","Type":"ContainerDied","Data":"f6fd9e9fe3f0bd6fe1a540dd62d4838fe69acfb540490943e3cd1e73ecd6b3e6"}
Oct 30 22:23:41.934338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:41.934223    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6fd9e9fe3f0bd6fe1a540dd62d4838fe69acfb540490943e3cd1e73ecd6b3e6"
Oct 30 22:23:42.071853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.071678    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" podStartSLOduration=3.013341488 podStartE2EDuration="21.07165362s" totalImagesPullingTime="18.058312132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:42.059069795 +0000 UTC m=+1352.908582791" watchObservedRunningTime="2026-10-30 22:23:42.07165362 +0000 UTC m=+1352.921166595"
Oct 30 22:23:42.208261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.208188    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:42.303472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.303442    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca\") pod \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") "
Oct 30 22:23:42.303714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.303674    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca\") pod \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") "
Oct 30 22:23:42.304077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.304050    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f\" (UniqueName: \"kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f\") pod \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\" (UID: \"b725fd6a-7e97-45ab-baec-a5a5c35970cf\") "
Oct 30 22:23:42.304353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.304238    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca" pod "b725fd6a-7e97-45ab-baec-a5a5c35970cf" (UID: "b725fd6a-7e97-45ab-baec-a5a5c35970cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:42.306863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.305192    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:42.309730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.309632    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f" pod "b725fd6a-7e97-45ab-baec-a5a5c35970cf" (UID: "b725fd6a-7e97-45ab-baec-a5a5c35970cf"). InnerVolumeSpecName "kube-api-access-lv55f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:42.315166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.313832    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca" pod "b725fd6a-7e97-45ab-baec-a5a5c35970cf" (UID: "b725fd6a-7e97-45ab-baec-a5a5c35970cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:42.315166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.314648    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"]
Oct 30 22:23:42.411885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.411727    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b725fd6a-7e97-45ab-baec-a5a5c35970cf-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:42.411885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.411760    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv55f\" (UniqueName: \"kubernetes.io/projected/b725fd6a-7e97-45ab-baec-a5a5c35970cf-kube-api-access-lv55f\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:42.868774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.868678    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:42.998257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.997430    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq" event={"ID":"b725fd6a-7e97-45ab-baec-a5a5c35970cf","Type":"ContainerDied","Data":"b6cfb83b71ff1d386748cb73081cb4fe538e16363b1ecb78aa59fe1db4c47011"}
Oct 30 22:23:42.998257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.997495    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6cfb83b71ff1d386748cb73081cb4fe538e16363b1ecb78aa59fe1db4c47011"
Oct 30 22:23:42.998257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:42.997575    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"
Oct 30 22:23:43.002890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.002834    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n" event={"ID":"ee1ae48e-31b2-48dc-a3a6-f00721ad662c","Type":"ContainerStarted","Data":"b86b133b817ee784e521f8c872aa7ff355d29a6cef780453c42f984e87031ec0"}
Oct 30 22:23:43.003076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.002914    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n" event={"ID":"ee1ae48e-31b2-48dc-a3a6-f00721ad662c","Type":"ContainerStarted","Data":"28d969b64904e6dfcdb7bf9f6457e23b49c06942c82f016578637bf5f4176bae"}
Oct 30 22:23:43.007351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.007302    5028 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qjq6b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:43.007516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.007361    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" podUID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:43.107381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.107333    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:43.124303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.124246    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n" podStartSLOduration=2.12423117 podStartE2EDuration="2.12423117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:43.11807625 +0000 UTC m=+1353.967589223" watchObservedRunningTime="2026-10-30 22:23:43.12423117 +0000 UTC m=+1353.973744143"
Oct 30 22:23:43.684478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.684444    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:43.878767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:43.878656    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:44.014026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:44.013989    5028 generic.go:356] "Generic (PLEG): container finished" podID="ee1ae48e-31b2-48dc-a3a6-f00721ad662c" containerID="b86b133b817ee784e521f8c872aa7ff355d29a6cef780453c42f984e87031ec0" exitCode=0
Oct 30 22:23:44.014321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:44.014128    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n" event={"ID":"ee1ae48e-31b2-48dc-a3a6-f00721ad662c","Type":"ContainerDied","Data":"b86b133b817ee784e521f8c872aa7ff355d29a6cef780453c42f984e87031ec0"}
Oct 30 22:23:45.212792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.212762    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:45.236905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.236863    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca\") pod \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") "
Oct 30 22:23:45.236905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.236912    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca\") pod \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") "
Oct 30 22:23:45.237132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.236950    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9\" (UniqueName: \"kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9\") pod \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\" (UID: \"ee1ae48e-31b2-48dc-a3a6-f00721ad662c\") "
Oct 30 22:23:45.237272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.237249    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca" pod "ee1ae48e-31b2-48dc-a3a6-f00721ad662c" (UID: "ee1ae48e-31b2-48dc-a3a6-f00721ad662c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:45.237579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.237560    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.239232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.239213    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9" pod "ee1ae48e-31b2-48dc-a3a6-f00721ad662c" (UID: "ee1ae48e-31b2-48dc-a3a6-f00721ad662c"). InnerVolumeSpecName "kube-api-access-kwtr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:45.239310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.239277    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca" pod "ee1ae48e-31b2-48dc-a3a6-f00721ad662c" (UID: "ee1ae48e-31b2-48dc-a3a6-f00721ad662c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:45.341746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.339185    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtr9\" (UniqueName: \"kubernetes.io/projected/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-kube-api-access-kwtr9\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:45.341746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:45.339223    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee1ae48e-31b2-48dc-a3a6-f00721ad662c-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:46.004263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:46.004229    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:23:46.037188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:46.037150    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"
Oct 30 22:23:46.037188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:46.037176    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n" event={"ID":"ee1ae48e-31b2-48dc-a3a6-f00721ad662c","Type":"ContainerDied","Data":"28d969b64904e6dfcdb7bf9f6457e23b49c06942c82f016578637bf5f4176bae"}
Oct 30 22:23:46.037431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:46.037232    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d969b64904e6dfcdb7bf9f6457e23b49c06942c82f016578637bf5f4176bae"
Oct 30 22:23:47.549279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:47.549241    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:49.282004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:49.281958    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:51.511722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.511670    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"]
Oct 30 22:23:51.512505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.512487    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ee1ae48e-31b2-48dc-a3a6-f00721ad662c" containerName="revoke-keys"
Oct 30 22:23:51.512560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.512506    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1ae48e-31b2-48dc-a3a6-f00721ad662c" containerName="revoke-keys"
Oct 30 22:23:51.512560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.512552    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b725fd6a-7e97-45ab-baec-a5a5c35970cf" containerName="revoke-keys"
Oct 30 22:23:51.512560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.512557    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b725fd6a-7e97-45ab-baec-a5a5c35970cf" containerName="revoke-keys"
Oct 30 22:23:51.512698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.512675    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b725fd6a-7e97-45ab-baec-a5a5c35970cf" containerName="revoke-keys"
Oct 30 22:23:51.512747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.512707    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1ae48e-31b2-48dc-a3a6-f00721ad662c" containerName="revoke-keys"
Oct 30 22:23:51.517214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.517193    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.621664 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.621629    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"]
Oct 30 22:23:51.644500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644454    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644542    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644622    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644655    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vpw\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644712    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644782    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644873    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.644988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.644977    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.645149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.645013    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.746785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.746746    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.746785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.746789    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vpw\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.746819    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.746870    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.746921    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747006    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747038    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747208    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747265    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747397    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747449    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747505    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.747675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.747648    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.748182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.748149    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.749646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.749624    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.749979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.749959    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.808301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.808228    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.808449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.808420    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vpw\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.835801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.835766    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:51.989246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:51.989214    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:52.274534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:52.274494    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"]
Oct 30 22:23:52.302738 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:23:52.302678    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632d78e1_94e3_4c66_bd57_7ecb64bd0c07.slice/crio-eefc8ec31310f15e1c6fc92c0f2fb0a3e22f203d040ef3756bb3ce7dbb2f4d09 WatchSource:0}: Error finding container eefc8ec31310f15e1c6fc92c0f2fb0a3e22f203d040ef3756bb3ce7dbb2f4d09: Status 404 returned error can't find the container with id eefc8ec31310f15e1c6fc92c0f2fb0a3e22f203d040ef3756bb3ce7dbb2f4d09
Oct 30 22:23:52.305522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:52.305484    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:52.305634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:52.305565    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:52.305634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:52.305598    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:53.156556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:53.156519    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" event={"ID":"632d78e1-94e3-4c66-bd57-7ecb64bd0c07","Type":"ContainerStarted","Data":"32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb"}
Oct 30 22:23:53.157044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:53.156577    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" event={"ID":"632d78e1-94e3-4c66-bd57-7ecb64bd0c07","Type":"ContainerStarted","Data":"eefc8ec31310f15e1c6fc92c0f2fb0a3e22f203d040ef3756bb3ce7dbb2f4d09"}
Oct 30 22:23:53.336968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:53.336904    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podStartSLOduration=2.33688561 podStartE2EDuration="2.33688561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:53.336229351 +0000 UTC m=+1364.185742324" watchObservedRunningTime="2026-10-30 22:23:53.33688561 +0000 UTC m=+1364.186398585"
Oct 30 22:23:53.841568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:53.841535    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:53.842000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:53.841971    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:23:53.842220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:53.842177    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 22:23:54.756425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:54.756395    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.837211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:54.837170    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:23:54.837474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:54.837448    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 22:23:54.932649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:54.932609    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:55.295367 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:55.295331    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:55.717584 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:55.717545    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:55.841712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:55.841205    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:23:55.841712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:55.841278    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 22:23:56.309543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:56.309506    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:56.565462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:56.565371    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:56.840170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:56.840083    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:23:56.840460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:56.840427    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 22:23:57.837709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:57.837172    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:23:57.837709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:57.837238    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 22:23:58.444909 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:23:58.444876    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:23:58.836490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:58.836456    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:23:58.836718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:58.836510    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 22:23:59.842725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:59.842678    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:59.843219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:59.843201    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:23:59.845866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:23:59.845840    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:24:01.909515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:01.909421    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:02.062624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:02.062584    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.732233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:05.732193    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.162574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:07.162526    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.309545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:07.309493    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:10.943622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:10.943576    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:11.099267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.099213    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:11.256298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.256246    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:11.566029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.565885    5028 scope.go:122] "RemoveContainer" containerID="ddbc67f4edd49c1fa274270f056b7b2cd49bdfe78cb09f17b4fe52a89eb8059c"
Oct 30 22:24:11.591464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.591442    5028 scope.go:122] "RemoveContainer" containerID="cb145b6592bf11c3314e38a95b36f4cda3cd02e99708bad92e673890d047b429"
Oct 30 22:24:11.604648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.604622    5028 scope.go:122] "RemoveContainer" containerID="d66ccfc4281ceae20093837b5db17bcbdf4e6154c67742f9ee0017468ad56f1c"
Oct 30 22:24:11.617794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.617767    5028 scope.go:122] "RemoveContainer" containerID="782c64074ebc2c4f9a5e10405a837c02c4baab0db6f47b713f862d64771b572c"
Oct 30 22:24:11.632276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.632243    5028 scope.go:122] "RemoveContainer" containerID="33a0316aefc2788c86975b95dd74eb8c9a038550d2446735708dd2b5f6f9b92e"
Oct 30 22:24:11.647599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.647572    5028 scope.go:122] "RemoveContainer" containerID="204625d9392954d9decd5e244a765b84837d8109d8f88291626bc7f8f0965d82"
Oct 30 22:24:11.663405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.663373    5028 scope.go:122] "RemoveContainer" containerID="da81eaa016ce52abad8f9e35c278c5deeb46eb0ca634b2188e6ad11772235b2f"
Oct 30 22:24:11.676906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.676872    5028 scope.go:122] "RemoveContainer" containerID="bf3c293dbe01276f7aced38003884f0ef881a43a690f84afa7cc2cbc5cd7573e"
Oct 30 22:24:11.691237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.691207    5028 scope.go:122] "RemoveContainer" containerID="ec5d65819ca30e8fe7c10346df57fecf9dc7a4b8976cc766a0e044b44196a8fe"
Oct 30 22:24:11.704761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.704663    5028 scope.go:122] "RemoveContainer" containerID="7db01f7831254dd631cbd866c302f4ff977988c4fb7d9d7d0039de53d6b8e9c3"
Oct 30 22:24:11.716865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.716842    5028 scope.go:122] "RemoveContainer" containerID="afde380fdf21af20b9b2696b63f52d7234da30827d46abc0c550fca0745f0cf7"
Oct 30 22:24:11.729607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.729578    5028 scope.go:122] "RemoveContainer" containerID="7ca3ea8b3518e9160c54f935f1bfbb17dbf7825023dcb31dfebfdbcc758a406f"
Oct 30 22:24:11.743370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:11.743344    5028 scope.go:122] "RemoveContainer" containerID="a96117ad2321852d4b148cfc2192fd6c958373bd9b64b01c58fb21eeffc17b94"
Oct 30 22:24:16.154305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:16.154269    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.598901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:20.598803    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.753364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:20.753298    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.918289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:20.918177    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:22.136815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:22.136766    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:25.425969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:25.425866    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.437251 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:24:28.437206    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:24:28.934801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:28.934767    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:30.034021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:30.033975    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.542935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:34.542894    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:39.145965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:39.145923    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.623399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:43.623360    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:48.356711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:48.347391    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:52.879152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:52.879057    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:56.845322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.845282    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"]
Oct 30 22:24:56.875866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.874522    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.984160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.984115    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.984471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.984442    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.984744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.984714    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cgs\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.985032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.984999    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.985192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.985162    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.985472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.985437    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.985646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.985622    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.985810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.985771    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:56.985973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:56.985944    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.001207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.001171    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"]
Oct 30 22:24:57.087427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087385    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cgs\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087518    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087545    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087657    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087731    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087758    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087783    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.087951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087909    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.088068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.087950    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.088609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.088583    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.088772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.088677    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.088884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.088805    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.089058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.089037    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.089503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.089478    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.091206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.091185    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.092205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.092179    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.147063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.146984    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cgs\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.147063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.147021    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.353600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.353561    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:57.697060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.697023    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:57.950781 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:24:57.928256    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb210a031_d3c3_444e_b75e_23816e31d3b0.slice/crio-e0bd0baabca2fdced34f813cb26b4e3390cf000ddff77d75774a5dceafbe1e7c WatchSource:0}: Error finding container e0bd0baabca2fdced34f813cb26b4e3390cf000ddff77d75774a5dceafbe1e7c: Status 404 returned error can't find the container with id e0bd0baabca2fdced34f813cb26b4e3390cf000ddff77d75774a5dceafbe1e7c
Oct 30 22:24:57.950781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.929239    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"]
Oct 30 22:24:57.950781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.930981    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:57.950781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.931045    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:57.950781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:57.931071    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:58.087028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:58.086994    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" event={"ID":"b210a031-d3c3-444e-b75e-23816e31d3b0","Type":"ContainerStarted","Data":"a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e"}
Oct 30 22:24:58.087226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:58.087061    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" event={"ID":"b210a031-d3c3-444e-b75e-23816e31d3b0","Type":"ContainerStarted","Data":"e0bd0baabca2fdced34f813cb26b4e3390cf000ddff77d75774a5dceafbe1e7c"}
Oct 30 22:24:58.221976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:58.221912    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podStartSLOduration=2.221892806 podStartE2EDuration="2.221892806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:58.2051043 +0000 UTC m=+1429.054617274" watchObservedRunningTime="2026-10-30 22:24:58.221892806 +0000 UTC m=+1429.071405779"
Oct 30 22:24:58.354381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:58.354349    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:24:58.485726 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:24:58.485615    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:24:59.102034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.102000    5028 generic.go:356] "Generic (PLEG): container finished" podID="36a8e4e9-b3dd-4338-8f65-160235583490" containerID="b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b" exitCode=0
Oct 30 22:24:59.102559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.102068    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" event={"ID":"36a8e4e9-b3dd-4338-8f65-160235583490","Type":"ContainerDied","Data":"b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"}
Oct 30 22:24:59.103031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.102997    5028 scope.go:122] "RemoveContainer" containerID="b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"
Oct 30 22:24:59.104930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.104901    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:59.105034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.104968    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:59.105034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.105001    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:59.173850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.173804    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"]
Oct 30 22:24:59.204451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.204422    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.270768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.265029    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpf8p\" (UniqueName: \"kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.279699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.274949    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.279699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.275074    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.291647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.280603    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:59.291647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.280705    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:59.291647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.280736    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:24:59.295777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.294160    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:59.385866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.380045    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:24:59.385866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.380117    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:24:59.385866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.382006    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpf8p\" (UniqueName: \"kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.385866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.382249    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.385866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.382306    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.385866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.383256    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.387499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.387414    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.407474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.406993    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"]
Oct 30 22:24:59.474093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.474036    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpf8p\" (UniqueName: \"kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p\") pod \"maas-api-revoke-sub-264c0a20-lhs2s\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:24:59.685521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:24:59.685426    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:25:00.148755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.146704    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" event={"ID":"36a8e4e9-b3dd-4338-8f65-160235583490","Type":"ContainerStarted","Data":"7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d"}
Oct 30 22:25:00.177142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.177114    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"]
Oct 30 22:25:00.238720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.235585    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:25:00.238720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.235775    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:00.238720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.235821    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:00.360711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.357388    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:00.360711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.357557    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:25:00.589860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.589535    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:25:00.589860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:00.589607    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:01.195928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.195833    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s" event={"ID":"d02b6505-66d7-4b97-bf34-149e4853f1f7","Type":"ContainerStarted","Data":"6a73b49e0a1a1af4f636c6d64b31f803711eedf51b6a8a7cd91120c7f6ce8943"}
Oct 30 22:25:01.196597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.196556    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s" event={"ID":"d02b6505-66d7-4b97-bf34-149e4853f1f7","Type":"ContainerStarted","Data":"d591f1b65d98fc8a32e1b164785f809a0af581d59af45bcc55e12f1ef9b6d7ef"}
Oct 30 22:25:01.246811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.235270    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:01.246811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.235335    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:01.315382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.315330    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s" podStartSLOduration=2.31531572 podStartE2EDuration="2.31531572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:01.310788226 +0000 UTC m=+1432.160301218" watchObservedRunningTime="2026-10-30 22:25:01.31531572 +0000 UTC m=+1432.164828719"
Oct 30 22:25:01.371711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.371296    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:01.371711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:01.371359    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:25:02.206561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.206528    5028 generic.go:356] "Generic (PLEG): container finished" podID="818ea594-fa7b-4161-b921-2888845dee5b" containerID="2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c" exitCode=0
Oct 30 22:25:02.207066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.206608    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" event={"ID":"818ea594-fa7b-4161-b921-2888845dee5b","Type":"ContainerDied","Data":"2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"}
Oct 30 22:25:02.207226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.207209    5028 scope.go:122] "RemoveContainer" containerID="2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"
Oct 30 22:25:02.209063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.209030    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:25:02.209162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.209110    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:25:02.209162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.209145    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:25:02.209613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.209561    5028 generic.go:356] "Generic (PLEG): container finished" podID="d02b6505-66d7-4b97-bf34-149e4853f1f7" containerID="6a73b49e0a1a1af4f636c6d64b31f803711eedf51b6a8a7cd91120c7f6ce8943" exitCode=0
Oct 30 22:25:02.209772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.209740    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s" event={"ID":"d02b6505-66d7-4b97-bf34-149e4853f1f7","Type":"ContainerDied","Data":"6a73b49e0a1a1af4f636c6d64b31f803711eedf51b6a8a7cd91120c7f6ce8943"}
Oct 30 22:25:02.231725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.225998    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:02.231725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.226061    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:02.391708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.391064    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:02.391708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.391125    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:25:02.608360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:02.608321    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:25:03.015192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.014305    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:03.240716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.237931    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:03.240716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.237995    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:03.265765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.265649    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" event={"ID":"818ea594-fa7b-4161-b921-2888845dee5b","Type":"ContainerStarted","Data":"b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d"}
Oct 30 22:25:03.389736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.377314    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:03.389736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.377377    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:25:03.502223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.502184    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"]
Oct 30 22:25:03.516085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.514484    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.602731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.601078    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.603191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.603162    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdb2\" (UniqueName: \"kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.603401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.603386    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.701712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.683903    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:25:03.725097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.725019    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.725321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.725296    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.778774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.776036    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdb2\" (UniqueName: \"kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.778774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.772032    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:25:03.778774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.776378    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:25:03.778774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.768175    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:03.815714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.813451    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:04.069772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:03.985990    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdb2\" (UniqueName: \"kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2\") pod \"maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:04.096836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.011191    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"]
Oct 30 22:25:04.171710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.110188    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:25:04.247724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.246895    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:04.247724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.246995    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:04.352257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.351958    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:04.390795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.385130    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca\") pod \"d02b6505-66d7-4b97-bf34-149e4853f1f7\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") "
Oct 30 22:25:04.390795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.385265    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p\" (UniqueName: \"kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p\") pod \"d02b6505-66d7-4b97-bf34-149e4853f1f7\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") "
Oct 30 22:25:04.390795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.385297    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca\") pod \"d02b6505-66d7-4b97-bf34-149e4853f1f7\" (UID: \"d02b6505-66d7-4b97-bf34-149e4853f1f7\") "
Oct 30 22:25:04.410406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.409867    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:04.410406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.409932    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 22:25:04.478623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.451650    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca" pod "d02b6505-66d7-4b97-bf34-149e4853f1f7" (UID: "d02b6505-66d7-4b97-bf34-149e4853f1f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:04.511519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.511480    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:04.567736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.555881    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p" pod "d02b6505-66d7-4b97-bf34-149e4853f1f7" (UID: "d02b6505-66d7-4b97-bf34-149e4853f1f7"). InnerVolumeSpecName "kube-api-access-hpf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:04.568154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.568114    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca" pod "d02b6505-66d7-4b97-bf34-149e4853f1f7" (UID: "d02b6505-66d7-4b97-bf34-149e4853f1f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:04.613829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.613420    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpf8p\" (UniqueName: \"kubernetes.io/projected/d02b6505-66d7-4b97-bf34-149e4853f1f7-kube-api-access-hpf8p\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:04.613829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.613453    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02b6505-66d7-4b97-bf34-149e4853f1f7-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:04.702635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.702590    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"
Oct 30 22:25:04.716899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.714097    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s" event={"ID":"d02b6505-66d7-4b97-bf34-149e4853f1f7","Type":"ContainerDied","Data":"d591f1b65d98fc8a32e1b164785f809a0af581d59af45bcc55e12f1ef9b6d7ef"}
Oct 30 22:25:04.716899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.714171    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d591f1b65d98fc8a32e1b164785f809a0af581d59af45bcc55e12f1ef9b6d7ef"
Oct 30 22:25:04.799748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.795893    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:25:04.799748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:04.795976    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:25:05.247719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.246702    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:05.247719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.246772    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:05.289487 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:25:05.289453    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a46f69_4003_4ed8_b174_275200ae25d5.slice/crio-c59c4874b35a1fce4a9cd589641ffd2f0191a753a67894c0f940a875c668d3cc WatchSource:0}: Error finding container c59c4874b35a1fce4a9cd589641ffd2f0191a753a67894c0f940a875c668d3cc: Status 404 returned error can't find the container with id c59c4874b35a1fce4a9cd589641ffd2f0191a753a67894c0f940a875c668d3cc
Oct 30 22:25:05.305300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.304994    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"]
Oct 30 22:25:05.392738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.392700    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:25:05.393029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.393007    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:25:05.394221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.394203    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:25:05.700713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.696147    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:25:05.700713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.696232    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:25:05.798509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:05.798415    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf" event={"ID":"b4a46f69-4003-4ed8-b174-275200ae25d5","Type":"ContainerStarted","Data":"c59c4874b35a1fce4a9cd589641ffd2f0191a753a67894c0f940a875c668d3cc"}
Oct 30 22:25:06.240763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:06.240245    5028 patch_prober.go:29] interesting pod/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:25:06.240763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:06.240315    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 22:25:06.664393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:06.664358    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:25:06.664814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:06.664413    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:25:06.809668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:06.809636    5028 generic.go:356] "Generic (PLEG): container finished" podID="b4a46f69-4003-4ed8-b174-275200ae25d5" containerID="5f6c079f80998556a23e26849fb4dd8bcf744d3d508613db6f9a2289f1e21f25" exitCode=0
Oct 30 22:25:06.809889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:06.809715    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf" event={"ID":"b4a46f69-4003-4ed8-b174-275200ae25d5","Type":"ContainerDied","Data":"5f6c079f80998556a23e26849fb4dd8bcf744d3d508613db6f9a2289f1e21f25"}
Oct 30 22:25:07.658535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:07.657147    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:07.663752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:07.663719    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:25:07.663932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:07.663767    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:25:07.722481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:07.722445    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:25:07.726323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:07.726302    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:25:08.150711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.150558    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:08.155568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.155536    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:08.204822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.204222    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca\") pod \"b4a46f69-4003-4ed8-b174-275200ae25d5\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") "
Oct 30 22:25:08.204822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.204284    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca\") pod \"b4a46f69-4003-4ed8-b174-275200ae25d5\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") "
Oct 30 22:25:08.204822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.204304    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2\" (UniqueName: \"kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2\") pod \"b4a46f69-4003-4ed8-b174-275200ae25d5\" (UID: \"b4a46f69-4003-4ed8-b174-275200ae25d5\") "
Oct 30 22:25:08.205166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.205102    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca" pod "b4a46f69-4003-4ed8-b174-275200ae25d5" (UID: "b4a46f69-4003-4ed8-b174-275200ae25d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:08.205626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.205598    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:08.209249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.209224    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca" pod "b4a46f69-4003-4ed8-b174-275200ae25d5" (UID: "b4a46f69-4003-4ed8-b174-275200ae25d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:08.209364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.209302    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2" pod "b4a46f69-4003-4ed8-b174-275200ae25d5" (UID: "b4a46f69-4003-4ed8-b174-275200ae25d5"). InnerVolumeSpecName "kube-api-access-5tdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:08.313271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.313226    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a46f69-4003-4ed8-b174-275200ae25d5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:08.313271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.313266    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdb2\" (UniqueName: \"kubernetes.io/projected/b4a46f69-4003-4ed8-b174-275200ae25d5-kube-api-access-5tdb2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:08.664310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.664272    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 22:25:08.664505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.664332    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 22:25:08.854356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.854320    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"
Oct 30 22:25:08.854356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.854341    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf" event={"ID":"b4a46f69-4003-4ed8-b174-275200ae25d5","Type":"ContainerDied","Data":"c59c4874b35a1fce4a9cd589641ffd2f0191a753a67894c0f940a875c668d3cc"}
Oct 30 22:25:08.854860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:08.854398    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59c4874b35a1fce4a9cd589641ffd2f0191a753a67894c0f940a875c668d3cc"
Oct 30 22:25:09.697746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:09.697673    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:25:09.698171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:09.698142    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:25:09.699188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:09.699162    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:25:09.790760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:09.790719    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:12.792256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.792223    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"]
Oct 30 22:25:12.793001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.792986    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b4a46f69-4003-4ed8-b174-275200ae25d5" containerName="revoke-keys"
Oct 30 22:25:12.793061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.793005    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a46f69-4003-4ed8-b174-275200ae25d5" containerName="revoke-keys"
Oct 30 22:25:12.793061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.793020    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d02b6505-66d7-4b97-bf34-149e4853f1f7" containerName="revoke-keys"
Oct 30 22:25:12.793061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.793025    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02b6505-66d7-4b97-bf34-149e4853f1f7" containerName="revoke-keys"
Oct 30 22:25:12.793161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.793146    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a46f69-4003-4ed8-b174-275200ae25d5" containerName="revoke-keys"
Oct 30 22:25:12.793161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.793155    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02b6505-66d7-4b97-bf34-149e4853f1f7" containerName="revoke-keys"
Oct 30 22:25:12.796062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.796040    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:12.852403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.852367    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"]
Oct 30 22:25:12.891660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.891630    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:12.891660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.891635    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:12.891660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.891635    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:12.893200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.893180    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:25:12.912146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.912115    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:12.912290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.912226    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gwd\" (UniqueName: \"kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:12.912382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:12.912360    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.014094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.014050    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.014303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.014165    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gwd\" (UniqueName: \"kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.014303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.014271    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.014965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.014929    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.016599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.016573    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.036620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.036569    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:13.093127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.093005    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gwd\" (UniqueName: \"kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd\") pod \"maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.110248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.110205    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:13.513786 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:25:13.513746    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa1b5fd_02d9_4d8d_a848_69e20fe7698e.slice/crio-93f2069da4cc65d555e664053b034af65bae4aff458c9fb405e82d62889d760d WatchSource:0}: Error finding container 93f2069da4cc65d555e664053b034af65bae4aff458c9fb405e82d62889d760d: Status 404 returned error can't find the container with id 93f2069da4cc65d555e664053b034af65bae4aff458c9fb405e82d62889d760d
Oct 30 22:25:13.516825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.516797    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"]
Oct 30 22:25:13.911223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:13.911183    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s" event={"ID":"caa1b5fd-02d9-4d8d-a848-69e20fe7698e","Type":"ContainerStarted","Data":"93f2069da4cc65d555e664053b034af65bae4aff458c9fb405e82d62889d760d"}
Oct 30 22:25:14.896996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:14.896902    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:14.924987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:14.924940    5028 generic.go:356] "Generic (PLEG): container finished" podID="caa1b5fd-02d9-4d8d-a848-69e20fe7698e" containerID="76f587fbcc6ba0a4cea9219f470552243f70383e0dd09e59034f53c1c56aea55" exitCode=0
Oct 30 22:25:14.925452 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:14.925119    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s" event={"ID":"caa1b5fd-02d9-4d8d-a848-69e20fe7698e","Type":"ContainerDied","Data":"76f587fbcc6ba0a4cea9219f470552243f70383e0dd09e59034f53c1c56aea55"}
Oct 30 22:25:15.117278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:15.117242    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:16.114715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.111316    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:16.654729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.653863    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:16.758439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.756972    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca\") pod \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") "
Oct 30 22:25:16.758439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.757043    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd\" (UniqueName: \"kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd\") pod \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") "
Oct 30 22:25:16.758439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.757111    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca\") pod \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\" (UID: \"caa1b5fd-02d9-4d8d-a848-69e20fe7698e\") "
Oct 30 22:25:16.758439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.758165    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca" pod "caa1b5fd-02d9-4d8d-a848-69e20fe7698e" (UID: "caa1b5fd-02d9-4d8d-a848-69e20fe7698e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:16.760230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.760201    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca" pod "caa1b5fd-02d9-4d8d-a848-69e20fe7698e" (UID: "caa1b5fd-02d9-4d8d-a848-69e20fe7698e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:16.760863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.760843    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd" pod "caa1b5fd-02d9-4d8d-a848-69e20fe7698e" (UID: "caa1b5fd-02d9-4d8d-a848-69e20fe7698e"). InnerVolumeSpecName "kube-api-access-v4gwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:16.859725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.859629    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.859725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.859662    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gwd\" (UniqueName: \"kubernetes.io/projected/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-kube-api-access-v4gwd\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.859725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.859673    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa1b5fd-02d9-4d8d-a848-69e20fe7698e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.946352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.946320    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"
Oct 30 22:25:16.946509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.946358    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s" event={"ID":"caa1b5fd-02d9-4d8d-a848-69e20fe7698e","Type":"ContainerDied","Data":"93f2069da4cc65d555e664053b034af65bae4aff458c9fb405e82d62889d760d"}
Oct 30 22:25:16.946509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:16.946420    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f2069da4cc65d555e664053b034af65bae4aff458c9fb405e82d62889d760d"
Oct 30 22:25:18.382751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:18.382718    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:20.095851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.095797    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"]
Oct 30 22:25:20.096647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.096627    5028 cpu_manager.go:414] "containerMap: removing container" podUID="caa1b5fd-02d9-4d8d-a848-69e20fe7698e" containerName="revoke-keys"
Oct 30 22:25:20.096739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.096648    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa1b5fd-02d9-4d8d-a848-69e20fe7698e" containerName="revoke-keys"
Oct 30 22:25:20.096848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.096835    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa1b5fd-02d9-4d8d-a848-69e20fe7698e" containerName="revoke-keys"
Oct 30 22:25:20.099760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.099740    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.160767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.159483    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"]
Oct 30 22:25:20.195252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.195217    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:20.195252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.195227    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:20.195503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.195322    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:20.197370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.197323    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:25:20.236990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.236955    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.237158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.237027    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.237227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.237201    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcbf\" (UniqueName: \"kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.339607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.339565    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.339872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.339648    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.339954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.339907    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcbf\" (UniqueName: \"kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.340774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.340744    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.342641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.342614    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.399221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.399151    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcbf\" (UniqueName: \"kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf\") pod \"maas-api-revoke-sub-7181a1b4-xw8hd\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.414035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.414002    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:20.803068 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:25:20.803038    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd360fdf6_9f19_437f_b963_721f2cc9b087.slice/crio-a4bc4e547f041a2bef99612bbe681507d2428ac19f605b95671527e2f42bc061 WatchSource:0}: Error finding container a4bc4e547f041a2bef99612bbe681507d2428ac19f605b95671527e2f42bc061: Status 404 returned error can't find the container with id a4bc4e547f041a2bef99612bbe681507d2428ac19f605b95671527e2f42bc061
Oct 30 22:25:20.806121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:20.806093    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"]
Oct 30 22:25:21.001251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:21.001215    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd" event={"ID":"d360fdf6-9f19-437f-b963-721f2cc9b087","Type":"ContainerStarted","Data":"a4bc4e547f041a2bef99612bbe681507d2428ac19f605b95671527e2f42bc061"}
Oct 30 22:25:22.015163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:22.015121    5028 generic.go:356] "Generic (PLEG): container finished" podID="d360fdf6-9f19-437f-b963-721f2cc9b087" containerID="1c3f3a1cb7b58591ebdc81d8d68ebd0810bc9e52601886ede0f682036f8de2a1" exitCode=0
Oct 30 22:25:22.015619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:22.015198    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd" event={"ID":"d360fdf6-9f19-437f-b963-721f2cc9b087","Type":"ContainerDied","Data":"1c3f3a1cb7b58591ebdc81d8d68ebd0810bc9e52601886ede0f682036f8de2a1"}
Oct 30 22:25:23.181242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.181202    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.267015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.266971    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:23.288415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.288311    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf\" (UniqueName: \"kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf\") pod \"d360fdf6-9f19-437f-b963-721f2cc9b087\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") "
Oct 30 22:25:23.288415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.288403    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca\") pod \"d360fdf6-9f19-437f-b963-721f2cc9b087\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") "
Oct 30 22:25:23.288806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.288781    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca\") pod \"d360fdf6-9f19-437f-b963-721f2cc9b087\" (UID: \"d360fdf6-9f19-437f-b963-721f2cc9b087\") "
Oct 30 22:25:23.289183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.289142    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca" pod "d360fdf6-9f19-437f-b963-721f2cc9b087" (UID: "d360fdf6-9f19-437f-b963-721f2cc9b087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:23.289561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.289541    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:23.291527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.291490    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca" pod "d360fdf6-9f19-437f-b963-721f2cc9b087" (UID: "d360fdf6-9f19-437f-b963-721f2cc9b087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:23.293024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.293000    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf" pod "d360fdf6-9f19-437f-b963-721f2cc9b087" (UID: "d360fdf6-9f19-437f-b963-721f2cc9b087"). InnerVolumeSpecName "kube-api-access-6qcbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:23.392385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.391537    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qcbf\" (UniqueName: \"kubernetes.io/projected/d360fdf6-9f19-437f-b963-721f2cc9b087-kube-api-access-6qcbf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:23.392385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.391575    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360fdf6-9f19-437f-b963-721f2cc9b087-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:23.434347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.434307    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"]
Oct 30 22:25:23.435943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.435897    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d360fdf6-9f19-437f-b963-721f2cc9b087" containerName="revoke-keys"
Oct 30 22:25:23.435943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.435924    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d360fdf6-9f19-437f-b963-721f2cc9b087" containerName="revoke-keys"
Oct 30 22:25:23.436227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.436139    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d360fdf6-9f19-437f-b963-721f2cc9b087" containerName="revoke-keys"
Oct 30 22:25:23.439992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.439964    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.492884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.492854    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"]
Oct 30 22:25:23.493221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.493192    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.493386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.493372    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.493553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.493533    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcrx\" (UniqueName: \"kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.595135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.595043    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.595294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.595182    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.595332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.595308    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcrx\" (UniqueName: \"kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.595781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.595760    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.597649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.597624    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.655100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.655068    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcrx\" (UniqueName: \"kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx\") pod \"maas-api-revoke-keys-e2e-ait-dee83803-7tgmb\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:23.756431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:23.756393    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:24.041918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:24.041882    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"
Oct 30 22:25:24.042078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:24.041889    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd" event={"ID":"d360fdf6-9f19-437f-b963-721f2cc9b087","Type":"ContainerDied","Data":"a4bc4e547f041a2bef99612bbe681507d2428ac19f605b95671527e2f42bc061"}
Oct 30 22:25:24.042078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:24.042021    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4bc4e547f041a2bef99612bbe681507d2428ac19f605b95671527e2f42bc061"
Oct 30 22:25:24.138243 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:25:24.138208    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaead273b_af62_4e49_ba87_63232ebcb4c0.slice/crio-68343935b761f159dbf32f91f08e3f345bdfdef520826db4b94c8a841d7b6a0a WatchSource:0}: Error finding container 68343935b761f159dbf32f91f08e3f345bdfdef520826db4b94c8a841d7b6a0a: Status 404 returned error can't find the container with id 68343935b761f159dbf32f91f08e3f345bdfdef520826db4b94c8a841d7b6a0a
Oct 30 22:25:24.140972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:24.140940    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"]
Oct 30 22:25:25.056846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:25.056808    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb" event={"ID":"aead273b-af62-4e49-ba87-63232ebcb4c0","Type":"ContainerStarted","Data":"8934d9f8f08f86adc47b822a944779fddc15e2885ceb68f9ecbc14c019d13233"}
Oct 30 22:25:25.057375 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:25.056867    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb" event={"ID":"aead273b-af62-4e49-ba87-63232ebcb4c0","Type":"ContainerStarted","Data":"68343935b761f159dbf32f91f08e3f345bdfdef520826db4b94c8a841d7b6a0a"}
Oct 30 22:25:25.169052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:25.168992    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb" podStartSLOduration=2.168975924 podStartE2EDuration="2.168975924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:25.168177 +0000 UTC m=+1456.017689973" watchObservedRunningTime="2026-10-30 22:25:25.168975924 +0000 UTC m=+1456.018488939"
Oct 30 22:25:25.782872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:25.782834    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:25.983723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:25.982847    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.070565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:26.070464    5028 generic.go:356] "Generic (PLEG): container finished" podID="aead273b-af62-4e49-ba87-63232ebcb4c0" containerID="8934d9f8f08f86adc47b822a944779fddc15e2885ceb68f9ecbc14c019d13233" exitCode=0
Oct 30 22:25:26.070976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:26.070554    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb" event={"ID":"aead273b-af62-4e49-ba87-63232ebcb4c0","Type":"ContainerDied","Data":"8934d9f8f08f86adc47b822a944779fddc15e2885ceb68f9ecbc14c019d13233"}
Oct 30 22:25:27.513877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.511469    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:27.577315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.577276    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx\" (UniqueName: \"kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx\") pod \"aead273b-af62-4e49-ba87-63232ebcb4c0\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") "
Oct 30 22:25:27.577468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.577391    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca\") pod \"aead273b-af62-4e49-ba87-63232ebcb4c0\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") "
Oct 30 22:25:27.577531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.577477    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca\") pod \"aead273b-af62-4e49-ba87-63232ebcb4c0\" (UID: \"aead273b-af62-4e49-ba87-63232ebcb4c0\") "
Oct 30 22:25:27.577879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.577846    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca" pod "aead273b-af62-4e49-ba87-63232ebcb4c0" (UID: "aead273b-af62-4e49-ba87-63232ebcb4c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:27.578306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.578285    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:27.579608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.579583    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx" pod "aead273b-af62-4e49-ba87-63232ebcb4c0" (UID: "aead273b-af62-4e49-ba87-63232ebcb4c0"). InnerVolumeSpecName "kube-api-access-twcrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:27.579708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.579651    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca" pod "aead273b-af62-4e49-ba87-63232ebcb4c0" (UID: "aead273b-af62-4e49-ba87-63232ebcb4c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:27.681741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.680506    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcrx\" (UniqueName: \"kubernetes.io/projected/aead273b-af62-4e49-ba87-63232ebcb4c0-kube-api-access-twcrx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:27.681741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:27.680547    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead273b-af62-4e49-ba87-63232ebcb4c0-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:28.131035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:28.131000    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"
Oct 30 22:25:28.131035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:28.131009    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb" event={"ID":"aead273b-af62-4e49-ba87-63232ebcb4c0","Type":"ContainerDied","Data":"68343935b761f159dbf32f91f08e3f345bdfdef520826db4b94c8a841d7b6a0a"}
Oct 30 22:25:28.131320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:28.131084    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68343935b761f159dbf32f91f08e3f345bdfdef520826db4b94c8a841d7b6a0a"
Oct 30 22:25:28.157607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:28.157573    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.444648 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:25:28.444559    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:25:28.859854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:28.859820    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.849385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:30.849249    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:32.747182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:32.746779    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.119008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:35.118901    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:25:35.119515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:35.119017    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:25:35.196205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:35.196169    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.375767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:35.373285    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.364807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:37.364753    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.624098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:37.623981    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:42.149888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:42.149777    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.080310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:46.080266    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:47.605508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:47.605464    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:47.876097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:47.876002    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:48.033352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.033317    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:48.258150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.258109    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"]
Oct 30 22:25:48.304846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.304813    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-05e3625f-mdkbf"]
Oct 30 22:25:48.367943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.367903    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"]
Oct 30 22:25:48.368234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.368179    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy" containerID="cri-o://b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d" gracePeriod=30
Oct 30 22:25:48.373139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.373114    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"]
Oct 30 22:25:48.373372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.373347    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" containerID="cri-o://a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e" gracePeriod=30
Oct 30 22:25:48.392757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:48.392716    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:49.080352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:49.080313    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:49.295481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:49.295371    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a46f69-4003-4ed8-b174-275200ae25d5" path="/var/lib/kubelet/pods/b4a46f69-4003-4ed8-b174-275200ae25d5/volumes"
Oct 30 22:25:49.407887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:49.407797    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:49.624917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:49.624874    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:50.080186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:50.080134    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:50.395880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:50.395799    5028 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:50.396411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:50.395882    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:50.629892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:50.629848    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.324833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:52.324793    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.483670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:52.483632    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:52.764232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:52.764196    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:53.644984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:53.644889    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:53.865813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:53.864657    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:53.905105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:53.905045    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:25:53.924434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:53.924405    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:25:54.040552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040519    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.040756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040605    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.040756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040651    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.040756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040722    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.040954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040760    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.040954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040823    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.040954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040864    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.040954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040914    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.040954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040911    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.040954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040925    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.040985    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041016    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041080    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041106    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041157    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041170    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041201    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.041269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041265    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041367    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy\") pod \"b210a031-d3c3-444e-b75e-23816e31d3b0\" (UID: \"b210a031-d3c3-444e-b75e-23816e31d3b0\") "
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041377    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041383    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041432    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041491    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041519    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert\") pod \"818ea594-fa7b-4161-b921-2888845dee5b\" (UID: \"818ea594-fa7b-4161-b921-2888845dee5b\") "
Oct 30 22:25:54.041701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.041579    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.042095 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042077    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:54.042442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042414    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:54.042620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042595    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042627    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b210a031-d3c3-444e-b75e-23816e31d3b0-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042643    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042657    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042671    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042715    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042730    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042743    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/818ea594-fa7b-4161-b921-2888845dee5b-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.042937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.042756    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.043275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.043250    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.044448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.044412    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "kube-api-access-p4cgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:54.045217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.045179    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.045314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.045225    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "kube-api-access-q7llc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:54.045314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.045290    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:54.045430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.045313    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token" pod "b210a031-d3c3-444e-b75e-23816e31d3b0" (UID: "b210a031-d3c3-444e-b75e-23816e31d3b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:54.045540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.045518    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.046132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.046116    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:54.046353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.046337    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo" pod "818ea594-fa7b-4161-b921-2888845dee5b" (UID: "818ea594-fa7b-4161-b921-2888845dee5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:54.146056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.145999    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/818ea594-fa7b-4161-b921-2888845dee5b-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146032    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/818ea594-fa7b-4161-b921-2888845dee5b-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146051    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146064    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7llc\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-kube-api-access-q7llc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146078    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/818ea594-fa7b-4161-b921-2888845dee5b-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146090    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4cgs\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-kube-api-access-p4cgs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146103    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b210a031-d3c3-444e-b75e-23816e31d3b0-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146116    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.146414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.146129    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b210a031-d3c3-444e-b75e-23816e31d3b0-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.441936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.441902    5028 generic.go:356] "Generic (PLEG): container finished" podID="818ea594-fa7b-4161-b921-2888845dee5b" containerID="b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d" exitCode=0
Oct 30 22:25:54.442136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.441975    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"
Oct 30 22:25:54.442136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.442009    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" event={"ID":"818ea594-fa7b-4161-b921-2888845dee5b","Type":"ContainerDied","Data":"b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d"}
Oct 30 22:25:54.442136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.442062    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h" event={"ID":"818ea594-fa7b-4161-b921-2888845dee5b","Type":"ContainerDied","Data":"833caa19177364889a09ce13589fb2fec969fdf3f04cf4ef286468dc5006b0b3"}
Oct 30 22:25:54.442136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.442092    5028 scope.go:122] "RemoveContainer" containerID="b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d"
Oct 30 22:25:54.452568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.452537    5028 generic.go:356] "Generic (PLEG): container finished" podID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerID="a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e" exitCode=0
Oct 30 22:25:54.452942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.452906    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" event={"ID":"b210a031-d3c3-444e-b75e-23816e31d3b0","Type":"ContainerDied","Data":"a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e"}
Oct 30 22:25:54.453052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.452972    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j" event={"ID":"b210a031-d3c3-444e-b75e-23816e31d3b0","Type":"ContainerDied","Data":"e0bd0baabca2fdced34f813cb26b4e3390cf000ddff77d75774a5dceafbe1e7c"}
Oct 30 22:25:54.453123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.453052    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"
Oct 30 22:25:54.466930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.466242    5028 scope.go:122] "RemoveContainer" containerID="2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"
Oct 30 22:25:54.479728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.479658    5028 scope.go:122] "RemoveContainer" containerID="b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d"
Oct 30 22:25:54.480103 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:25:54.480075    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d\": container with ID starting with b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d not found: ID does not exist" containerID="b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d"
Oct 30 22:25:54.480211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.480111    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d"} err="failed to get container status \"b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d\": rpc error: code = NotFound desc = could not find container \"b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d\": container with ID starting with b4601e8d135501e7022c5fae6e931e87aa1b27565793f92cf918ad485386422d not found: ID does not exist"
Oct 30 22:25:54.480211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.480131    5028 scope.go:122] "RemoveContainer" containerID="2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"
Oct 30 22:25:54.480437 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:25:54.480413    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c\": container with ID starting with 2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c not found: ID does not exist" containerID="2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"
Oct 30 22:25:54.480514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.480447    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c"} err="failed to get container status \"2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c\": rpc error: code = NotFound desc = could not find container \"2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c\": container with ID starting with 2ad2fada9f4d31baef650a95a169fe291ed0cb7f1bd54d75ca4a6e95ae330a5c not found: ID does not exist"
Oct 30 22:25:54.480514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.480469    5028 scope.go:122] "RemoveContainer" containerID="a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e"
Oct 30 22:25:54.491447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.491419    5028 scope.go:122] "RemoveContainer" containerID="a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e"
Oct 30 22:25:54.491845 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:25:54.491818    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e\": container with ID starting with a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e not found: ID does not exist" containerID="a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e"
Oct 30 22:25:54.491987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.491852    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e"} err="failed to get container status \"a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e\": rpc error: code = NotFound desc = could not find container \"a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e\": container with ID starting with a3fc44c79026eace387e44826ba2b2609a02a348a02d32f8382674b52b2a0f8e not found: ID does not exist"
Oct 30 22:25:54.581156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.581118    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.607893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.607853    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"]
Oct 30 22:25:54.639090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.639049    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-v2f2j"]
Oct 30 22:25:54.740831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.740797    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"]
Oct 30 22:25:54.747373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:54.747344    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-6h48h"]
Oct 30 22:25:55.256302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:55.256263    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:55.299356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:55.299321    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818ea594-fa7b-4161-b921-2888845dee5b" path="/var/lib/kubelet/pods/818ea594-fa7b-4161-b921-2888845dee5b/volumes"
Oct 30 22:25:55.299937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:55.299922    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" path="/var/lib/kubelet/pods/b210a031-d3c3-444e-b75e-23816e31d3b0/volumes"
Oct 30 22:25:55.567889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:55.567804    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:55.781049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:55.781015    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"]
Oct 30 22:25:55.827704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:55.827598    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae20a62d-lmx6s"]
Oct 30 22:25:56.121937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:56.121845    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.303517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:56.303483    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.069709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.069566    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"]
Oct 30 22:25:57.109944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.109908    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e826-7zkb4"]
Oct 30 22:25:57.280164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.279117    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"]
Oct 30 22:25:57.280164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.279398    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy" containerID="cri-o://7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d" gracePeriod=30
Oct 30 22:25:57.297893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.297858    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b89f52-b35e-4364-b77a-1d966eeb6874" path="/var/lib/kubelet/pods/83b89f52-b35e-4364-b77a-1d966eeb6874/volumes"
Oct 30 22:25:57.298503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.298481    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa1b5fd-02d9-4d8d-a848-69e20fe7698e" path="/var/lib/kubelet/pods/caa1b5fd-02d9-4d8d-a848-69e20fe7698e/volumes"
Oct 30 22:25:57.319371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.319329    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.991757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:57.991710    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:58.503709 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:25:58.503385    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:25:58.648543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.647447    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:58.996163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.996124    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"]
Oct 30 22:25:58.997511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.997480    5028 cpu_manager.go:414] "containerMap: removing container" podUID="aead273b-af62-4e49-ba87-63232ebcb4c0" containerName="revoke-keys"
Oct 30 22:25:58.997732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.997715    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aead273b-af62-4e49-ba87-63232ebcb4c0" containerName="revoke-keys"
Oct 30 22:25:58.997877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.997865    5028 cpu_manager.go:414] "containerMap: removing container" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy"
Oct 30 22:25:58.997970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.997957    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy"
Oct 30 22:25:58.998119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.998108    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy"
Oct 30 22:25:58.998198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.998188    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy"
Oct 30 22:25:58.998460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.998445    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy"
Oct 30 22:25:58.998573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.998563    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b210a031-d3c3-444e-b75e-23816e31d3b0" containerName="istio-proxy"
Oct 30 22:25:58.998668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:58.998657    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="aead273b-af62-4e49-ba87-63232ebcb4c0" containerName="revoke-keys"
Oct 30 22:25:59.012917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.012887    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.020526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.020402    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:59.048272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.048236    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"]
Oct 30 22:25:59.117819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.117780    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.117862    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m492g\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118084    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118141    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118203    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118406    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118468    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118572    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.118646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.118611    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.220652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.220612    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.220864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.220765    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.220864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.220822    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.220978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.220882    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.220978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.220911    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.220990    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221044    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m492g\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221226    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221266    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221304    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221515    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221744    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.221845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.221778    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.222045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.222019    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.224130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.224048    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.225103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.225077    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.280237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.280147    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.280405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.280381    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m492g\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g\") pod \"maas-default-gateway-openshift-default-75bf867b6c-l597h\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.324160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.324119    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:59.349152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.348276    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:25:59.861030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.860994    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"]
Oct 30 22:25:59.863596 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:25:59.863565    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b99c592_742b_4b7d_a50e_70e079c7741e.slice/crio-15d1d3748f6e5aea6c78a166533783f40ea24496e4913283dc8d7131d3bbc84c WatchSource:0}: Error finding container 15d1d3748f6e5aea6c78a166533783f40ea24496e4913283dc8d7131d3bbc84c: Status 404 returned error can't find the container with id 15d1d3748f6e5aea6c78a166533783f40ea24496e4913283dc8d7131d3bbc84c
Oct 30 22:25:59.867005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.866973    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:25:59.867103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.867040    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:25:59.867103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.867073    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:25:59.908908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:25:59.908865    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:00.088735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.084771    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:00.466837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.466798    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:00.659638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.659131    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" event={"ID":"8b99c592-742b-4b7d-a50e-70e079c7741e","Type":"ContainerStarted","Data":"b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500"}
Oct 30 22:26:00.659638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.659315    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" event={"ID":"8b99c592-742b-4b7d-a50e-70e079c7741e","Type":"ContainerStarted","Data":"15d1d3748f6e5aea6c78a166533783f40ea24496e4913283dc8d7131d3bbc84c"}
Oct 30 22:26:00.675089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.675054    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:00.883841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.880193    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podStartSLOduration=2.8801722180000002 podStartE2EDuration="2.880172218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:00.830260625 +0000 UTC m=+1491.679773643" watchObservedRunningTime="2026-10-30 22:26:00.880172218 +0000 UTC m=+1491.729685191"
Oct 30 22:26:00.942722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:00.942287    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:01.116233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.115560    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:01.293576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.288514    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:01.454739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.449279    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:01.497992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.497621    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:01.497992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.497717    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:01.524709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.519304    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"]
Oct 30 22:26:01.524709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.519357    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:26:01.524709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.519399    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dee83803-7tgmb"]
Oct 30 22:26:01.762969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:01.761325    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.353806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.353118    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:02.353806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.353205    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:02.659618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.659587    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:26:02.701743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.701670    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.701947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.701761    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.701947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.701833    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.701947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.701924    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.702141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.701988    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.702141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702099    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.702141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702122    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:02.702304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702148    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:02.702304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702198    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.702304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702246    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.702304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702279    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9\") pod \"36a8e4e9-b3dd-4338-8f65-160235583490\" (UID: \"36a8e4e9-b3dd-4338-8f65-160235583490\") "
Oct 30 22:26:02.702541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702299    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:02.702541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702443    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:02.702643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702540    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:02.703012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.702994    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.703111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703017    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a8e4e9-b3dd-4338-8f65-160235583490-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.703111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703031    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.703111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703045    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.703111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703058    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.703827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703797    5028 generic.go:356] "Generic (PLEG): container finished" podID="36a8e4e9-b3dd-4338-8f65-160235583490" containerID="7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d" exitCode=0
Oct 30 22:26:02.704026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703838    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" event={"ID":"36a8e4e9-b3dd-4338-8f65-160235583490","Type":"ContainerDied","Data":"7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d"}
Oct 30 22:26:02.704026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703895    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"
Oct 30 22:26:02.704026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703916    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6" event={"ID":"36a8e4e9-b3dd-4338-8f65-160235583490","Type":"ContainerDied","Data":"62d91cdcb1dcead7f46a8f34b0479efeacf2a3e494aa6abe41bc35b65935b81a"}
Oct 30 22:26:02.704026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.703948    5028 scope.go:122] "RemoveContainer" containerID="7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d"
Oct 30 22:26:02.705806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.705760    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:02.705806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.705781    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:02.706148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.706122    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "kube-api-access-9z8z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:02.706393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.706368    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token" pod "36a8e4e9-b3dd-4338-8f65-160235583490" (UID: "36a8e4e9-b3dd-4338-8f65-160235583490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:02.759178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.759139    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.768568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.768543    5028 scope.go:122] "RemoveContainer" containerID="b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"
Oct 30 22:26:02.783670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.783647    5028 scope.go:122] "RemoveContainer" containerID="7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d"
Oct 30 22:26:02.784060 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:26:02.784027    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d\": container with ID starting with 7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d not found: ID does not exist" containerID="7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d"
Oct 30 22:26:02.784193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.784065    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d"} err="failed to get container status \"7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d\": rpc error: code = NotFound desc = could not find container \"7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d\": container with ID starting with 7b34ea8d9bb9d422132cd15f968af8c954c906633c00a086aefef7f31bb9770d not found: ID does not exist"
Oct 30 22:26:02.784193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.784084    5028 scope.go:122] "RemoveContainer" containerID="b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"
Oct 30 22:26:02.784386 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:26:02.784363    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b\": container with ID starting with b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b not found: ID does not exist" containerID="b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"
Oct 30 22:26:02.784460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.784390    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b"} err="failed to get container status \"b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b\": rpc error: code = NotFound desc = could not find container \"b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b\": container with ID starting with b5168584fb95e1c352eca3c1291ed41a57dbf8e4f72065552da1431eb143ce0b not found: ID does not exist"
Oct 30 22:26:02.804559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.804526    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a8e4e9-b3dd-4338-8f65-160235583490-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.804794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.804771    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8z9\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-kube-api-access-9z8z9\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.804794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.804794    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a8e4e9-b3dd-4338-8f65-160235583490-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.804964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.804804    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a8e4e9-b3dd-4338-8f65-160235583490-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:02.913154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:02.913071    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:03.093475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.093430    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:03.150466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.150419    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"]
Oct 30 22:26:03.157915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.157881    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ae20a62d-openshift-default-77d6577fcf-qdsn6"]
Oct 30 22:26:03.261601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.261568    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:03.299788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.297082    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" path="/var/lib/kubelet/pods/36a8e4e9-b3dd-4338-8f65-160235583490/volumes"
Oct 30 22:26:03.299788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.298044    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aead273b-af62-4e49-ba87-63232ebcb4c0" path="/var/lib/kubelet/pods/aead273b-af62-4e49-ba87-63232ebcb4c0/volumes"
Oct 30 22:26:03.350133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.350092    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:03.350455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:03.350406    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:04.354969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:04.351974    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:04.354969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:04.352036    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:05.350823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:05.350784    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:05.351125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:05.351092    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:06.258512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.258460    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.353488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.349864    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:06.353488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.349933    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:06.493831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.493397    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:06.693962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.693859    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"]
Oct 30 22:26:06.695154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695124    5028 cpu_manager.go:414] "containerMap: removing container" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy"
Oct 30 22:26:06.695154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695151    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy"
Oct 30 22:26:06.695368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695185    5028 cpu_manager.go:414] "containerMap: removing container" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy"
Oct 30 22:26:06.695368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695193    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy"
Oct 30 22:26:06.695368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695205    5028 cpu_manager.go:414] "containerMap: removing container" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy"
Oct 30 22:26:06.695368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695212    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy"
Oct 30 22:26:06.695621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695385    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ea594-fa7b-4161-b921-2888845dee5b" containerName="istio-proxy"
Oct 30 22:26:06.695621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695403    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy"
Oct 30 22:26:06.695621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.695411    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a8e4e9-b3dd-4338-8f65-160235583490" containerName="istio-proxy"
Oct 30 22:26:06.702565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.702533    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:06.750921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.750885    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"]
Oct 30 22:26:06.795074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.794991    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e8ed1149-dockercfg-dxr4k\""
Oct 30 22:26:06.813099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.813068    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e8ed1149\""
Oct 30 22:26:06.826726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.823418    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:06.826726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.823728    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkr8j\" (UniqueName: \"kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j\") pod \"payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:06.864395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.864341    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"]
Oct 30 22:26:06.870980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.870945    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:06.885899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.885868    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"]
Oct 30 22:26:06.926744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.926706    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:06.926947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.926854    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkr8j\" (UniqueName: \"kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j\") pod \"payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:06.929417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:06.929384    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:07.027707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.020921    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkr8j\" (UniqueName: \"kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j\") pod \"payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:07.027707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.022450    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:07.036704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.034265    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume\") pod \"payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") " pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.040826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.038528    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kkm\" (UniqueName: \"kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm\") pod \"payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") " pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.142351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.140523    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kkm\" (UniqueName: \"kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm\") pod \"payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") " pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.142351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.140935    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume\") pod \"payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") " pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.142351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.142006    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume\") pod \"payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") " pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.215469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.215376    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kkm\" (UniqueName: \"kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm\") pod \"payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") " pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.232513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.232234    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:07.428763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.416888    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:07.456856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.439434    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:07.560712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.560669    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"]
Oct 30 22:26:07.688016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.687935    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:07.799815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.799784    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"]
Oct 30 22:26:07.850787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.850738    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx" event={"ID":"29be3106-bfa3-45ad-bacc-5ae7bac0b29d","Type":"ContainerStarted","Data":"50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565"}
Oct 30 22:26:07.851255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.850868    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx" event={"ID":"29be3106-bfa3-45ad-bacc-5ae7bac0b29d","Type":"ContainerStarted","Data":"b4a8dd95dde5e4b88d927f33fbb635c8a049b83170c8354ee41f7730ddb82078"}
Oct 30 22:26:07.851346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.851293    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:07.852867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.852838    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv" event={"ID":"ece83cc6-b519-468e-a2dd-f33432a3ca34","Type":"ContainerStarted","Data":"b4355627675fc61fe7ee538b874fb551d54c17e5a1f3bce5ed62e3b7b4707546"}
Oct 30 22:26:07.962508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.962454    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx" podStartSLOduration=1.962435616 podStartE2EDuration="1.962435616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:07.960710594 +0000 UTC m=+1498.810223564" watchObservedRunningTime="2026-10-30 22:26:07.962435616 +0000 UTC m=+1498.811948590"
Oct 30 22:26:07.995718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:07.995663    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.153997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.153959    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.310997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.310909    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.349237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.349199    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:26:08.349397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.349264    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 22:26:08.463661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.463629    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.867923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.867883    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv" event={"ID":"ece83cc6-b519-468e-a2dd-f33432a3ca34","Type":"ContainerStarted","Data":"14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a"}
Oct 30 22:26:08.868115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.868011    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:08.985877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:08.985821    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv" podStartSLOduration=2.985805337 podStartE2EDuration="2.985805337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:08.981344605 +0000 UTC m=+1499.830857603" watchObservedRunningTime="2026-10-30 22:26:08.985805337 +0000 UTC m=+1499.835318309"
Oct 30 22:26:09.639833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:09.639803    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:26:09.643025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:09.642990    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:26:09.766269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:09.766232    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:26:09.766924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:09.766835    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:26:09.768071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:09.768036    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:26:11.270558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.270507    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:11.606772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.604583    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:11.697580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.697528    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"]
Oct 30 22:26:11.704820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.704790    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.751711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.732855    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtwf\" (UniqueName: \"kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf\") pod \"payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.751711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.733259    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.794709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.794066    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-926fc6\""
Oct 30 22:26:11.794709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.794297    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-926fc6-dockercfg-ld8bc\""
Oct 30 22:26:11.820711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.805971    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"]
Oct 30 22:26:11.885874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.869105    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.885874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.869200    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtwf\" (UniqueName: \"kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf\") pod \"payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.940380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.940150    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtwf\" (UniqueName: \"kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf\") pod \"payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.941194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.941145    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:11.990711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:11.970456    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"]
Oct 30 22:26:12.018719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.012301    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"]
Oct 30 22:26:12.018719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.012425    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.049864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.042836    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:12.109761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.109712    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9z2w\" (UniqueName: \"kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w\") pod \"payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.110054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.110036    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.232665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.216235    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.234593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.234054    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9z2w\" (UniqueName: \"kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w\") pod \"payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.234593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.231788    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.398530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.398472    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9z2w\" (UniqueName: \"kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w\") pod \"payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.463711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.459152    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:12.855333 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:12.855295    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0f919c_bad2_48f0_8b52_a70234598d71.slice/crio-5724c8b4902a8eba7b8f15b78c44fc1363c918e063b8f6254cc4a2b7daea5930 WatchSource:0}: Error finding container 5724c8b4902a8eba7b8f15b78c44fc1363c918e063b8f6254cc4a2b7daea5930: Status 404 returned error can't find the container with id 5724c8b4902a8eba7b8f15b78c44fc1363c918e063b8f6254cc4a2b7daea5930
Oct 30 22:26:12.857487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.857459    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"]
Oct 30 22:26:12.930027 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:12.929993    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d431a0_67bb_469a_ae0c_b672ae71f7fd.slice/crio-7df743739363e86a06ec9499f88a629763d5c50ef3435397bbc2f5e59e973591 WatchSource:0}: Error finding container 7df743739363e86a06ec9499f88a629763d5c50ef3435397bbc2f5e59e973591: Status 404 returned error can't find the container with id 7df743739363e86a06ec9499f88a629763d5c50ef3435397bbc2f5e59e973591
Oct 30 22:26:12.933735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:12.933679    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"]
Oct 30 22:26:13.423556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.423515    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh" event={"ID":"68d431a0-67bb-469a-ae0c-b672ae71f7fd","Type":"ContainerStarted","Data":"fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484"}
Oct 30 22:26:13.424141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.423594    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh" event={"ID":"68d431a0-67bb-469a-ae0c-b672ae71f7fd","Type":"ContainerStarted","Data":"7df743739363e86a06ec9499f88a629763d5c50ef3435397bbc2f5e59e973591"}
Oct 30 22:26:13.424141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.423624    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:13.426010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.425973    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh" event={"ID":"6a0f919c-bad2-48f0-8b52-a70234598d71","Type":"ContainerStarted","Data":"e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa"}
Oct 30 22:26:13.426136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.426029    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh" event={"ID":"6a0f919c-bad2-48f0-8b52-a70234598d71","Type":"ContainerStarted","Data":"5724c8b4902a8eba7b8f15b78c44fc1363c918e063b8f6254cc4a2b7daea5930"}
Oct 30 22:26:13.426136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.426059    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:13.538012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.537956    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh" podStartSLOduration=2.5379400370000003 podStartE2EDuration="2.537940037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:13.534406943 +0000 UTC m=+1504.383919916" watchObservedRunningTime="2026-10-30 22:26:13.537940037 +0000 UTC m=+1504.387453061"
Oct 30 22:26:13.577413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.577330    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.652563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.652490    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh" podStartSLOduration=2.652471999 podStartE2EDuration="2.652471999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:13.645361461 +0000 UTC m=+1504.494874482" watchObservedRunningTime="2026-10-30 22:26:13.652471999 +0000 UTC m=+1504.501984973"
Oct 30 22:26:13.733730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:13.733668    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.014470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.014385    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"]
Oct 30 22:26:14.019928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.019899    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.131281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.130033    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"]
Oct 30 22:26:14.192221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192182    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192368    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192400    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192483    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192525    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192633    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192803    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.192916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192901    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48v7\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.193010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.192989    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294386    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294423    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294455    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294483    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294529    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294636    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294718    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48v7\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.294794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294785    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.295054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294816    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.295054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294879    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.295054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.294995    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.295054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.295024    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.295270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.295245    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.295524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.295505    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.297128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.297102    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.297207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.297101    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.356360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.356328    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.356729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.356669    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48v7\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nwxll\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:14.639832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:14.639739    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:15.075652 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:15.075620    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cd3361_d3a9_4147_b1c4_5cddb8037c69.slice/crio-96664bf6aa906d3ab32cc69c67faa6b6906a3ecc962bd4da1d1c8a00c2fb3f3e WatchSource:0}: Error finding container 96664bf6aa906d3ab32cc69c67faa6b6906a3ecc962bd4da1d1c8a00c2fb3f3e: Status 404 returned error can't find the container with id 96664bf6aa906d3ab32cc69c67faa6b6906a3ecc962bd4da1d1c8a00c2fb3f3e
Oct 30 22:26:15.079185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.079142    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:15.079282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.079235    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:15.079319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.079280    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:15.079917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.079857    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"]
Oct 30 22:26:15.148240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.148212    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.467943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.465960    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" event={"ID":"c4cd3361-d3a9-4147-b1c4-5cddb8037c69","Type":"ContainerStarted","Data":"5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37"}
Oct 30 22:26:15.467943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.466044    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" event={"ID":"c4cd3361-d3a9-4147-b1c4-5cddb8037c69","Type":"ContainerStarted","Data":"96664bf6aa906d3ab32cc69c67faa6b6906a3ecc962bd4da1d1c8a00c2fb3f3e"}
Oct 30 22:26:15.587954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.587888    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podStartSLOduration=2.587865591 podStartE2EDuration="2.587865591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:15.586240715 +0000 UTC m=+1506.435753688" watchObservedRunningTime="2026-10-30 22:26:15.587865591 +0000 UTC m=+1506.437378562"
Oct 30 22:26:15.640161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:15.640123    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:16.434247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:16.432599    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:16.654284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:16.653579    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:16.660752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:16.658563    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:16.660752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:16.658627    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:17.640715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:17.640188    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:17.640715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:17.640256    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:18.597255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:18.597213    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:18.657717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:18.655101    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:18.657717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:18.655173    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:18.871720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:18.871603    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:26:19.640879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:19.640844    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:19.641492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:19.641459    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:19.885638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:19.885596    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:26:20.642172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:20.642136    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:20.642709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:20.642272    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:20.915880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:20.915793    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:21.141311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:21.141273    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:21.641077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:21.641041    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:21.641269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:21.641101    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:22.640882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:22.640845    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 22:26:22.641349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:22.640919    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 22:26:23.574104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:23.574061    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:24.001740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.001658    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:24.230307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.230262    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:24.266569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.266434    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:24.270172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.270142    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:24.271067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.271041    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:26:24.428771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.428728    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"]
Oct 30 22:26:24.433952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.433929    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.442064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.442039    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:26:24.442359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.442171    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:26:24.490711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.489888    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-7c5d18\""
Oct 30 22:26:24.490711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.490046    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-7c5d18-dockercfg-j9zmp\""
Oct 30 22:26:24.497391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.497361    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"]
Oct 30 22:26:24.577188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.577095    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529bt\" (UniqueName: \"kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt\") pod \"payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.577957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.577927    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.604380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.604330    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"]
Oct 30 22:26:24.615524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.615469    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.680877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.679943    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.680877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.680074    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcd9s\" (UniqueName: \"kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s\") pod \"payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.680877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.680139    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.680877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.680331    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529bt\" (UniqueName: \"kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt\") pod \"payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.685969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.681975    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.701174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.701096    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"]
Oct 30 22:26:24.766099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.765612    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-529bt\" (UniqueName: \"kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt\") pod \"payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.777780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.776940    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:24.782297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.781679    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcd9s\" (UniqueName: \"kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s\") pod \"payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.782297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.781751    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.784371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.784312    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.870814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.854816    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcd9s\" (UniqueName: \"kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s\") pod \"payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:24.967765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:24.962063    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:25.554557 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:25.554515    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0056b572_af70_46a3_8959_e6136b357831.slice/crio-dd05b627a3298881c4a61c218a0d72b67e48fdb905490ecf706d1b50708fc3bb WatchSource:0}: Error finding container dd05b627a3298881c4a61c218a0d72b67e48fdb905490ecf706d1b50708fc3bb: Status 404 returned error can't find the container with id dd05b627a3298881c4a61c218a0d72b67e48fdb905490ecf706d1b50708fc3bb
Oct 30 22:26:25.559640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.559609    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"]
Oct 30 22:26:25.598990 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:25.598875    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc51319e_8440_42eb_a477_b1cff7ac94bf.slice/crio-ef84e6489c4d9b4244d6721dcc621c7cfea2787df1e6ff77c049de07f4159338 WatchSource:0}: Error finding container ef84e6489c4d9b4244d6721dcc621c7cfea2787df1e6ff77c049de07f4159338: Status 404 returned error can't find the container with id ef84e6489c4d9b4244d6721dcc621c7cfea2787df1e6ff77c049de07f4159338
Oct 30 22:26:25.602129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.602100    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"]
Oct 30 22:26:25.728712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.728662    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld" event={"ID":"0056b572-af70-46a3-8959-e6136b357831","Type":"ContainerStarted","Data":"027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b"}
Oct 30 22:26:25.728915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.728736    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld" event={"ID":"0056b572-af70-46a3-8959-e6136b357831","Type":"ContainerStarted","Data":"dd05b627a3298881c4a61c218a0d72b67e48fdb905490ecf706d1b50708fc3bb"}
Oct 30 22:26:25.728915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.728768    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:25.730883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.730856    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl" event={"ID":"fc51319e-8440-42eb-a477-b1cff7ac94bf","Type":"ContainerStarted","Data":"3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf"}
Oct 30 22:26:25.731024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.730898    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl" event={"ID":"fc51319e-8440-42eb-a477-b1cff7ac94bf","Type":"ContainerStarted","Data":"ef84e6489c4d9b4244d6721dcc621c7cfea2787df1e6ff77c049de07f4159338"}
Oct 30 22:26:25.731024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.730994    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:25.842549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.842445    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld" podStartSLOduration=1.842430268 podStartE2EDuration="1.842430268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:25.840722614 +0000 UTC m=+1516.690235586" watchObservedRunningTime="2026-10-30 22:26:25.842430268 +0000 UTC m=+1516.691943241"
Oct 30 22:26:25.957785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:25.957310    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl" podStartSLOduration=1.957294716 podStartE2EDuration="1.957294716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:25.955051307 +0000 UTC m=+1516.804564281" watchObservedRunningTime="2026-10-30 22:26:25.957294716 +0000 UTC m=+1516.806807768"
Oct 30 22:26:26.054070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:26.054033    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"]
Oct 30 22:26:26.069170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:26.068173    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d38b00-gbfpl"]
Oct 30 22:26:27.295352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:27.295319    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c44e6b4-178b-4df3-9500-4191238d4f1f" path="/var/lib/kubelet/pods/9c44e6b4-178b-4df3-9500-4191238d4f1f/volumes"
Oct 30 22:26:28.437597 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:26:28.437559    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:26:28.747389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:28.747355    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:33.238614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:33.238574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.743751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:36.743716    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:26:36.744166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:36.743807    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:26:37.051912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:37.051819    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"]
Oct 30 22:26:37.058187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:37.058152    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137c9bdd-njvcd"]
Oct 30 22:26:37.294414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:37.294379    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c204a1-a5a4-4788-a86d-1e8854c46e7a" path="/var/lib/kubelet/pods/76c204a1-a5a4-4788-a86d-1e8854c46e7a/volumes"
Oct 30 22:26:37.675459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:37.675419    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.829375 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:37.829335    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:38.053981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:38.053947    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:39.948297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:39.948259    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:40.103175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:40.103137    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:40.406105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:40.406065    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.185273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:42.185231    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:46.758177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:46.758128    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:46.918297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:46.918257    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:47.138136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:47.138031    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:47.324946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:47.324905    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.199119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.199081    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.743259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.743226    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"]
Oct 30 22:26:48.750012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.749983    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.795880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.795845    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtdt\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.796292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.796265    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.796476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.796460    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.796742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.796618    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.796995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.796975    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.797116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.797101    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.797230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.797216    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.797335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.797320    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.797552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.797531    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.832707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.828699    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-29bbe735-openshift-default-dockercfg-sgjmc\""
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.956905    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982134    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982183    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982224    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982253    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982346    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982385    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtdt\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982552    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982630    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.982718    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.984348    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.984624    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.984924    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.986460    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.987410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.987027    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:48.991462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:48.991433    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:49.015320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.015205    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:49.062704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.062644    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"]
Oct 30 22:26:49.066906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.066841    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtdt\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:49.073305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.073247    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:49.093998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.093961    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:49.400784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.394610    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:49.587418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.587374    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:49.587511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.587468    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:49.587550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.587508    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:49.588197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:49.588169    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"]
Oct 30 22:26:50.100361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:50.100318    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" event={"ID":"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5","Type":"ContainerStarted","Data":"8c1b43780b2afafc3df4d78b7fd05bc8ca91bb2e49bf3f9b25dd8ec79cfbe3d0"}
Oct 30 22:26:50.100572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:50.100385    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" event={"ID":"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5","Type":"ContainerStarted","Data":"8e22895d7d1e78c82f41ace1c81873dcc5cf245b396a17847dc94efae302f266"}
Oct 30 22:26:50.107577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:50.107536    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:50.109773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:50.109743    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:26:50.109891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:50.109797    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 22:26:50.222362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:50.222307    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podStartSLOduration=2.222291371 podStartE2EDuration="2.222291371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:50.21720001 +0000 UTC m=+1541.066712989" watchObservedRunningTime="2026-10-30 22:26:50.222291371 +0000 UTC m=+1541.071804344"
Oct 30 22:26:51.057722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:51.057646    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"]
Oct 30 22:26:51.104596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:51.104559    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9f3095-vchsx"]
Oct 30 22:26:51.111951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:51.111921    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:51.295814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:51.295783    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf486d15-fa6a-4e26-b54a-e4eed5865fd2" path="/var/lib/kubelet/pods/bf486d15-fa6a-4e26-b54a-e4eed5865fd2/volumes"
Oct 30 22:26:51.681646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:51.681615    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:52.120652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:52.120611    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:52.121632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:52.121606    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:26:53.058509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:53.058460    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"]
Oct 30 22:26:53.098579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:53.098544    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f77f7123-2mmxr"]
Oct 30 22:26:53.175995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:53.175964    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.298408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:53.298372    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a9c68d-f18b-4954-b59a-eede185a5a43" path="/var/lib/kubelet/pods/99a9c68d-f18b-4954-b59a-eede185a5a43/volumes"
Oct 30 22:26:53.354896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:53.354797    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.525157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:53.525118    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:56.419792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.419754    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:56.573274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.573232    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:56.749563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.749526    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:56.841584 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.841544    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"]
Oct 30 22:26:56.852483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.852453    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:56.909320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.909273    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"]
Oct 30 22:26:56.931873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.931824    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:56.932172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.932150    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fv4\" (UniqueName: \"kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:56.938627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.938591    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-29bbe735\""
Oct 30 22:26:56.938835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:56.938815    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-29bbe735-dockercfg-jq5hf\""
Oct 30 22:26:57.097770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.097451    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fv4\" (UniqueName: \"kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:57.097955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.097852    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:57.100720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.099051    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:57.138208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.138164    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"]
Oct 30 22:26:57.158716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.157920    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"]
Oct 30 22:26:57.158716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.158080    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.207950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.206241    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"]
Oct 30 22:26:57.215768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.215727    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fv4\" (UniqueName: \"kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:57.220487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.220333    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f27beb53-hfg2g"]
Oct 30 22:26:57.303538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.303493    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f1f6ce-d8c5-48e1-a3de-99f714f2e45a" path="/var/lib/kubelet/pods/07f1f6ce-d8c5-48e1-a3de-99f714f2e45a/volumes"
Oct 30 22:26:57.303962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.303833    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.304354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.304328    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zc4\" (UniqueName: \"kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.405826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.405726    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.406024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.405995    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc4\" (UniqueName: \"kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.406540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.406508    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.465668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.465630    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zc4\" (UniqueName: \"kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.472963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.472938    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:57.531817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.531735    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:57.862201 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:57.862166    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc808208d_bc25_45d2_8c33_33c1d4823300.slice/crio-e332562d9ebb9893855735adab0e33e7672705e2e2493fcc14535b53033adb87 WatchSource:0}: Error finding container e332562d9ebb9893855735adab0e33e7672705e2e2493fcc14535b53033adb87: Status 404 returned error can't find the container with id e332562d9ebb9893855735adab0e33e7672705e2e2493fcc14535b53033adb87
Oct 30 22:26:57.864430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.864405    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"]
Oct 30 22:26:57.965984 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:26:57.965953    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b49da8_b3c0_46fa_8168_904081b0ad22.slice/crio-dba7c5ee17c02fb37a24ec6990e7689c5108fcd034988482bbbba2a46a0a270d WatchSource:0}: Error finding container dba7c5ee17c02fb37a24ec6990e7689c5108fcd034988482bbbba2a46a0a270d: Status 404 returned error can't find the container with id dba7c5ee17c02fb37a24ec6990e7689c5108fcd034988482bbbba2a46a0a270d
Oct 30 22:26:57.968921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:57.968894    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"]
Oct 30 22:26:58.240119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.240077    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" event={"ID":"f3b49da8-b3c0-46fa-8168-904081b0ad22","Type":"ContainerStarted","Data":"ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42"}
Oct 30 22:26:58.240325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.240139    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" event={"ID":"f3b49da8-b3c0-46fa-8168-904081b0ad22","Type":"ContainerStarted","Data":"dba7c5ee17c02fb37a24ec6990e7689c5108fcd034988482bbbba2a46a0a270d"}
Oct 30 22:26:58.240325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.240165    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:26:58.242402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.242375    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" event={"ID":"c808208d-bc25-45d2-8c33-33c1d4823300","Type":"ContainerStarted","Data":"248e78baca2da53758f111de9cd20730afb7ed8b862909bff9bc347c5af5f0b5"}
Oct 30 22:26:58.242549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.242425    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" event={"ID":"c808208d-bc25-45d2-8c33-33c1d4823300","Type":"ContainerStarted","Data":"e332562d9ebb9893855735adab0e33e7672705e2e2493fcc14535b53033adb87"}
Oct 30 22:26:58.242549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.242477    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:26:58.353805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.353752    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" podStartSLOduration=2.353737921 podStartE2EDuration="2.353737921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:58.351909784 +0000 UTC m=+1549.201422757" watchObservedRunningTime="2026-10-30 22:26:58.353737921 +0000 UTC m=+1549.203250895"
Oct 30 22:26:58.444416 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:26:58.444370    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:26:58.470800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:26:58.470748    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" podStartSLOduration=2.470733226 podStartE2EDuration="2.470733226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:58.463986248 +0000 UTC m=+1549.313499221" watchObservedRunningTime="2026-10-30 22:26:58.470733226 +0000 UTC m=+1549.320246198"
Oct 30 22:27:01.142892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.142859    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:01.728314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.728205    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"]
Oct 30 22:27:01.735980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.735948    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.817718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.811100    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-61346ee8-openshift-default-dockercfg-s852m\""
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.818496    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.818560    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.818643    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.819032    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdtb\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.819101    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.819149    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.819306    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.819677    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.823737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.819816    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.838471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.838426    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:01.978415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.957352    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.978826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.978657    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:01.979135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.979114    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdtb\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.001251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:01.985697    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"]
Oct 30 22:27:02.001552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.000863    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:02.003370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.001849    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.003737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.003721    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.004097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.004078    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.004897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.004881    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.005388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.005370    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.006365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.004449    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.006612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.004829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.007101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.006205    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.007236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.007057    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.007427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.007413    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.007550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.007538    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.038877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.038841    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.045408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.045362    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.084910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.076377    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.084910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.078023    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdtb\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb\") pod \"e2e-mt-61346ee8-openshift-default-c79649cc-r8659\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") " pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.266251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.266145    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:02.365648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.365602    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:02.824936 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:27:02.824903    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e6dd8e_8361_4624_9ff2_7cc06b2f11e7.slice/crio-cde607ae9e3f6025d92d73f664494b00981af46b2307578b8e63625c62991d95 WatchSource:0}: Error finding container cde607ae9e3f6025d92d73f664494b00981af46b2307578b8e63625c62991d95: Status 404 returned error can't find the container with id cde607ae9e3f6025d92d73f664494b00981af46b2307578b8e63625c62991d95
Oct 30 22:27:02.827186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.827160    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"]
Oct 30 22:27:02.827905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.827873    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:27:02.827992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.827953    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:27:02.828067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:02.827997    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:27:03.303121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:03.303082    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" event={"ID":"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7","Type":"ContainerStarted","Data":"e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"}
Oct 30 22:27:03.303525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:03.303145    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" event={"ID":"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7","Type":"ContainerStarted","Data":"cde607ae9e3f6025d92d73f664494b00981af46b2307578b8e63625c62991d95"}
Oct 30 22:27:03.366594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:03.366559    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:03.371952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:03.371923    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:03.427827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:03.427771    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podStartSLOduration=2.427753831 podStartE2EDuration="2.427753831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:03.418625578 +0000 UTC m=+1554.268138553" watchObservedRunningTime="2026-10-30 22:27:03.427753831 +0000 UTC m=+1554.277266816"
Oct 30 22:27:04.314174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:04.314137    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:04.315385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:04.315362    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:27:06.220241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.220205    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.359110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.359071    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"]
Oct 30 22:27:06.359380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.359351    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy" containerID="cri-o://e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b" gracePeriod=30
Oct 30 22:27:06.385659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.385626    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.403767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.403736    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"]
Oct 30 22:27:06.403949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.403929    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" containerID="cri-o://71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a" gracePeriod=30
Oct 30 22:27:06.410187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.410160    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"]
Oct 30 22:27:06.410406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.410376    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy" containerID="cri-o://862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47" gracePeriod=30
Oct 30 22:27:06.451539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.451503    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"]
Oct 30 22:27:06.451800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.451759    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" containerID="cri-o://e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72" gracePeriod=30
Oct 30 22:27:06.560137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.560032    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.723037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.721891    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:06.723037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.721956    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:06.733315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:06.733276    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:08.052659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:08.052620    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"]
Oct 30 22:27:08.098201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:08.098148    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4513b1-7dzr4"]
Oct 30 22:27:08.363942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:08.363855    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:09.098871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.098835    5028 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:09.099266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.098894    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:09.254320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.254289    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:27:09.254773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.254742    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:27:09.299369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.299334    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc337aa3-a1e7-4b2c-a87e-368a02c09c66" path="/var/lib/kubelet/pods/dc337aa3-a1e7-4b2c-a87e-368a02c09c66/volumes"
Oct 30 22:27:09.345889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.345853    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:09.510914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:09.510873    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:10.104711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:10.094560    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"]
Oct 30 22:27:10.131725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:10.128861    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f53c404-wqn8d"]
Oct 30 22:27:11.294090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.294058    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5b4300-b174-4466-8a8c-9ce221bb6a89" path="/var/lib/kubelet/pods/6e5b4300-b174-4466-8a8c-9ce221bb6a89/volumes"
Oct 30 22:27:11.732626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.725843    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:27:11.838105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.838013    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.838105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.838062    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.838105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.838100    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.838401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.838157    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.838401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.838189    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.838991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.838959    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:11.839393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.839367    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.839498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.839448    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.839582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.839566    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.839649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.839632    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token\") pod \"2474dd98-1521-4f25-b10d-675ac3157105\" (UID: \"2474dd98-1521-4f25-b10d-675ac3157105\") "
Oct 30 22:27:11.840475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.840448    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.842555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.842400    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:11.842555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.842484    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:11.842555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.842534    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:11.843138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.843118    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:11.854264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.854178    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:11.878848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.878807    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:11.879402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.879365    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "kube-api-access-cfgt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:11.879553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.879470    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:11.888791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.886275    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token" pod "2474dd98-1521-4f25-b10d-675ac3157105" (UID: "2474dd98-1521-4f25-b10d-675ac3157105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946556    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946592    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgt9\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-kube-api-access-cfgt9\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946610    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2474dd98-1521-4f25-b10d-675ac3157105-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946623    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2474dd98-1521-4f25-b10d-675ac3157105-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946637    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946651    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.957742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946665    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2474dd98-1521-4f25-b10d-675ac3157105-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.963854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.946678    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2474dd98-1521-4f25-b10d-675ac3157105-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:11.971768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.969963    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:27:11.983806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:11.981616    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:27:12.003110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.003079    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163227    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163288    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163322    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163371    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163404    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163468    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163501    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163559    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163593    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163624    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163708    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163745    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163807    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163842    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163902    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.163957    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164017    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164072    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164139    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164179    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164208    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164236    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164306    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert\") pod \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\" (UID: \"106fddfb-ab86-488e-9d28-31a4b45a7bdc\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164354    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164393    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164447    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy\") pod \"e660d4aa-1866-4615-a66b-b5d3d499d00b\" (UID: \"e660d4aa-1866-4615-a66b-b5d3d499d00b\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164540    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data\") pod \"b40e1899-0698-432a-9c93-784212968f3f\" (UID: \"b40e1899-0698-432a-9c93-784212968f3f\") "
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.164614    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.165469    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.165495    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.165774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.165736    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.167737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.166214    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.167737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.166745    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.167737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.166962    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.168118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168095    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:12.168201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168164    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:12.168201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168178    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.168309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168208    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.169776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168417    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.169776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168562    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.169776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168717    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.169776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.168857    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.169776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.169127    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:12.177576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.177522    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:12.180896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.180678    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.195835    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "kube-api-access-pxhnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.195867    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.195890    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.195913    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "kube-api-access-5ddcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.195965    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.196013    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy" pod "e660d4aa-1866-4615-a66b-b5d3d499d00b" (UID: "e660d4aa-1866-4615-a66b-b5d3d499d00b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.196123    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.196243    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.196347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.196313    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token" pod "b40e1899-0698-432a-9c93-784212968f3f" (UID: "b40e1899-0698-432a-9c93-784212968f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.198855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.198048    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2" pod "106fddfb-ab86-488e-9d28-31a4b45a7bdc" (UID: "106fddfb-ab86-488e-9d28-31a4b45a7bdc"). InnerVolumeSpecName "kube-api-access-28zg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:12.267243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267202    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40e1899-0698-432a-9c93-784212968f3f-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267242    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267256    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267272    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddcw\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-kube-api-access-5ddcw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267288    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267302    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267317    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267329    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267345    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zg2\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-kube-api-access-28zg2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267359    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40e1899-0698-432a-9c93-784212968f3f-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267372    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267385    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267400    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhnj\" (UniqueName: \"kubernetes.io/projected/e660d4aa-1866-4615-a66b-b5d3d499d00b-kube-api-access-pxhnj\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267413    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267442    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267455    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267468    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.267478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267482    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267495    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e660d4aa-1866-4615-a66b-b5d3d499d00b-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267508    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267522    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40e1899-0698-432a-9c93-784212968f3f-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267534    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e660d4aa-1866-4615-a66b-b5d3d499d00b-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267547    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e660d4aa-1866-4615-a66b-b5d3d499d00b-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267561    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40e1899-0698-432a-9c93-784212968f3f-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267574    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.268222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.267588    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106fddfb-ab86-488e-9d28-31a4b45a7bdc-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:12.422755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.422646    5028 generic.go:356] "Generic (PLEG): container finished" podID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerID="e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72" exitCode=0
Oct 30 22:27:12.422755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.422732    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"
Oct 30 22:27:12.423259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.422722    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" event={"ID":"e660d4aa-1866-4615-a66b-b5d3d499d00b","Type":"ContainerDied","Data":"e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72"}
Oct 30 22:27:12.423259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.422957    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn" event={"ID":"e660d4aa-1866-4615-a66b-b5d3d499d00b","Type":"ContainerDied","Data":"dad44ae6d9d8a23116c7bdb5351179bf4176440ba9d07da43ce0fc9b7f2d2488"}
Oct 30 22:27:12.423259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.422994    5028 scope.go:122] "RemoveContainer" containerID="e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72"
Oct 30 22:27:12.426223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.426195    5028 generic.go:356] "Generic (PLEG): container finished" podID="b40e1899-0698-432a-9c93-784212968f3f" containerID="862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47" exitCode=0
Oct 30 22:27:12.426461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.426257    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"
Oct 30 22:27:12.426537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.426277    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" event={"ID":"b40e1899-0698-432a-9c93-784212968f3f","Type":"ContainerDied","Data":"862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47"}
Oct 30 22:27:12.426595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.426556    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q" event={"ID":"b40e1899-0698-432a-9c93-784212968f3f","Type":"ContainerDied","Data":"6abca6e9a3bf848f2bccee8da2c4b9a613c5839d853c2c3c23e6d275ea881205"}
Oct 30 22:27:12.428832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.428805    5028 generic.go:356] "Generic (PLEG): container finished" podID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerID="71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a" exitCode=0
Oct 30 22:27:12.429090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.429071    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" event={"ID":"106fddfb-ab86-488e-9d28-31a4b45a7bdc","Type":"ContainerDied","Data":"71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a"}
Oct 30 22:27:12.429189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.429118    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v" event={"ID":"106fddfb-ab86-488e-9d28-31a4b45a7bdc","Type":"ContainerDied","Data":"75b9b0a66e056867ebb7ad8ee93192755048c6a95d461a1767fd5cd8e1c18f8d"}
Oct 30 22:27:12.429776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.429756    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"
Oct 30 22:27:12.431420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.431343    5028 generic.go:356] "Generic (PLEG): container finished" podID="2474dd98-1521-4f25-b10d-675ac3157105" containerID="e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b" exitCode=0
Oct 30 22:27:12.431515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.431472    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" event={"ID":"2474dd98-1521-4f25-b10d-675ac3157105","Type":"ContainerDied","Data":"e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b"}
Oct 30 22:27:12.431581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.431514    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js" event={"ID":"2474dd98-1521-4f25-b10d-675ac3157105","Type":"ContainerDied","Data":"d7e1ed949633a6a7edd827ad9bba6cb39c2c330eefd65987e28abe7c95616714"}
Oct 30 22:27:12.431647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.431585    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"
Oct 30 22:27:12.458618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.458587    5028 scope.go:122] "RemoveContainer" containerID="32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"
Oct 30 22:27:12.486027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.486002    5028 scope.go:122] "RemoveContainer" containerID="e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72"
Oct 30 22:27:12.486349 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:12.486326    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72\": container with ID starting with e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72 not found: ID does not exist" containerID="e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72"
Oct 30 22:27:12.486436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.486357    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72"} err="failed to get container status \"e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72\": rpc error: code = NotFound desc = could not find container \"e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72\": container with ID starting with e09f57e09bbecd38d42969919f5ae710146366680a23bd67569bdfe6c8b65c72 not found: ID does not exist"
Oct 30 22:27:12.486436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.486374    5028 scope.go:122] "RemoveContainer" containerID="32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"
Oct 30 22:27:12.486639 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:12.486620    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc\": container with ID starting with 32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc not found: ID does not exist" containerID="32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"
Oct 30 22:27:12.486699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.486645    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc"} err="failed to get container status \"32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc\": rpc error: code = NotFound desc = could not find container \"32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc\": container with ID starting with 32e846cecd3f345305b03c1cf08a577e8ea99cb6f70082befa125b0d85b4acbc not found: ID does not exist"
Oct 30 22:27:12.486699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.486658    5028 scope.go:122] "RemoveContainer" containerID="862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47"
Oct 30 22:27:12.498669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.498638    5028 scope.go:122] "RemoveContainer" containerID="862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47"
Oct 30 22:27:12.499160 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:12.499110    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47\": container with ID starting with 862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47 not found: ID does not exist" containerID="862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47"
Oct 30 22:27:12.499233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.499183    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47"} err="failed to get container status \"862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47\": rpc error: code = NotFound desc = could not find container \"862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47\": container with ID starting with 862df5b0e3f3377d4cdeeaf7cc4718358fb70ae950f99d4efd58306d3c2edb47 not found: ID does not exist"
Oct 30 22:27:12.499233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.499214    5028 scope.go:122] "RemoveContainer" containerID="71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a"
Oct 30 22:27:12.511180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.511156    5028 scope.go:122] "RemoveContainer" containerID="71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a"
Oct 30 22:27:12.511482 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:12.511460    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a\": container with ID starting with 71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a not found: ID does not exist" containerID="71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a"
Oct 30 22:27:12.511555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.511491    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a"} err="failed to get container status \"71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a\": rpc error: code = NotFound desc = could not find container \"71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a\": container with ID starting with 71a7b7dc9d7b87f71b569ade7fccf2517928cc482194af02c04470a1bdd0fd4a not found: ID does not exist"
Oct 30 22:27:12.511555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.511508    5028 scope.go:122] "RemoveContainer" containerID="e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b"
Oct 30 22:27:12.522998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.522855    5028 scope.go:122] "RemoveContainer" containerID="e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b"
Oct 30 22:27:12.523248 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:12.523210    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b\": container with ID starting with e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b not found: ID does not exist" containerID="e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b"
Oct 30 22:27:12.523296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.523260    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b"} err="failed to get container status \"e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b\": rpc error: code = NotFound desc = could not find container \"e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b\": container with ID starting with e97aec63a58fcc2a34d55c329c3307c84292276622083772534d320905625e1b not found: ID does not exist"
Oct 30 22:27:12.542858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.542831    5028 scope.go:122] "RemoveContainer" containerID="4cc186c5326ca96dc9278cfcd5d82193549b6c6771852fec6f3e2c41bd6cc66e"
Oct 30 22:27:12.587725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.587673    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"]
Oct 30 22:27:12.595804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.595776    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-dj9js"]
Oct 30 22:27:12.720764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.699247    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"]
Oct 30 22:27:12.720764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.708296    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-bcrmn"]
Oct 30 22:27:12.862949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.862912    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"]
Oct 30 22:27:12.869570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.869542    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-v844q"]
Oct 30 22:27:12.974318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.974229    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"]
Oct 30 22:27:12.980168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:12.980136    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-t792v"]
Oct 30 22:27:13.301767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:13.301675    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" path="/var/lib/kubelet/pods/106fddfb-ab86-488e-9d28-31a4b45a7bdc/volumes"
Oct 30 22:27:13.302248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:13.302234    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2474dd98-1521-4f25-b10d-675ac3157105" path="/var/lib/kubelet/pods/2474dd98-1521-4f25-b10d-675ac3157105/volumes"
Oct 30 22:27:13.302751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:13.302736    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40e1899-0698-432a-9c93-784212968f3f" path="/var/lib/kubelet/pods/b40e1899-0698-432a-9c93-784212968f3f/volumes"
Oct 30 22:27:13.303566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:13.303551    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" path="/var/lib/kubelet/pods/e660d4aa-1866-4615-a66b-b5d3d499d00b/volumes"
Oct 30 22:27:17.132502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:17.132459    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.310327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:17.310262    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.474292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:17.474247    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:18.052421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:18.052387    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"]
Oct 30 22:27:18.098416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:18.098376    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4338d8-7x5kw"]
Oct 30 22:27:19.306930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:19.306722    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b9de13-8797-42f8-89c0-e01761292eb5" path="/var/lib/kubelet/pods/68b9de13-8797-42f8-89c0-e01761292eb5/volumes"
Oct 30 22:27:20.053621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:20.053579    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"]
Oct 30 22:27:20.100235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:20.100197    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9036ba6-9grsk"]
Oct 30 22:27:21.055803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:21.055761    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"]
Oct 30 22:27:21.100820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:21.100781    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f698129e-svz2j"]
Oct 30 22:27:21.294291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:21.294257    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363c2e60-14b3-461d-9c00-75d2af311c33" path="/var/lib/kubelet/pods/363c2e60-14b3-461d-9c00-75d2af311c33/volumes"
Oct 30 22:27:21.294885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:21.294870    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6b5cc7-0bf3-4143-a6d6-029ce057c870" path="/var/lib/kubelet/pods/bf6b5cc7-0bf3-4143-a6d6-029ce057c870/volumes"
Oct 30 22:27:22.381999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:22.381958    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:24.051252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:24.051213    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"]
Oct 30 22:27:24.098129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:24.098091    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300f715d-xvkss"]
Oct 30 22:27:25.051460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:25.051424    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"]
Oct 30 22:27:25.098849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:25.098813    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b86ed724-c9bqk"]
Oct 30 22:27:25.293565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:25.293531    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9947f535-998c-4fb2-b6e9-f98fae674a36" path="/var/lib/kubelet/pods/9947f535-998c-4fb2-b6e9-f98fae674a36/volumes"
Oct 30 22:27:25.294119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:25.294102    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ae36f2-05bc-4e3f-b23e-351b4dcc069b" path="/var/lib/kubelet/pods/f7ae36f2-05bc-4e3f-b23e-351b4dcc069b/volumes"
Oct 30 22:27:26.822138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:26.822102    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:26.989163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:26.989109    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:28.064414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:28.064379    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"]
Oct 30 22:27:28.111413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:28.111377    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98951e5-rgx7l"]
Oct 30 22:27:28.437146 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:28.437050    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:27:29.051542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:29.051506    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"]
Oct 30 22:27:29.098458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:29.098423    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d47f0567-zwrnk"]
Oct 30 22:27:29.293919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:29.293886    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47" path="/var/lib/kubelet/pods/261995b5-8ff5-48ce-b0d2-a3c4f3bf0e47/volumes"
Oct 30 22:27:29.294387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:29.294330    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c0158a-8d75-4f4c-bfbf-7fea099c1192" path="/var/lib/kubelet/pods/84c0158a-8d75-4f4c-bfbf-7fea099c1192/volumes"
Oct 30 22:27:31.712749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:31.712669    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:33.144801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:33.144762    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:36.301524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:36.301485    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:39.051666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:39.051628    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"]
Oct 30 22:27:39.098986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:39.098949    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db53485-nc4kv"]
Oct 30 22:27:39.293358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:39.293314    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a044fb0a-059b-4716-9b65-60deab08a2e5" path="/var/lib/kubelet/pods/a044fb0a-059b-4716-9b65-60deab08a2e5/volumes"
Oct 30 22:27:40.642902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:40.642845    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:41.052226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:41.052186    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"]
Oct 30 22:27:41.099120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:41.099083    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755c74df-8vtzp"]
Oct 30 22:27:41.295604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:41.295573    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e589c9e-80fa-46d8-a78d-bd60e2120611" path="/var/lib/kubelet/pods/8e589c9e-80fa-46d8-a78d-bd60e2120611/volumes"
Oct 30 22:27:43.054517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:43.054474    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"]
Oct 30 22:27:43.097969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:43.097932    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e4c92f-cgvnf"]
Oct 30 22:27:43.294391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:43.294358    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190a8ef4-20c7-466f-83ed-f0afa1d14370" path="/var/lib/kubelet/pods/190a8ef4-20c7-466f-83ed-f0afa1d14370/volumes"
Oct 30 22:27:45.489115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:45.489074    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:46.222229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.222192    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"]
Oct 30 22:27:46.223133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223097    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy"
Oct 30 22:27:46.223133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223132    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy"
Oct 30 22:27:46.223347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223212    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy"
Oct 30 22:27:46.223347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223220    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy"
Oct 30 22:27:46.223347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223239    5028 cpu_manager.go:414] "containerMap: removing container" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy"
Oct 30 22:27:46.223347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223246    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy"
Oct 30 22:27:46.223347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223264    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy"
Oct 30 22:27:46.223347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223271    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy"
Oct 30 22:27:46.223639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223468    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy"
Oct 30 22:27:46.223639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223482    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2474dd98-1521-4f25-b10d-675ac3157105" containerName="istio-proxy"
Oct 30 22:27:46.223639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223493    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="106fddfb-ab86-488e-9d28-31a4b45a7bdc" containerName="istio-proxy"
Oct 30 22:27:46.223639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223504    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40e1899-0698-432a-9c93-784212968f3f" containerName="istio-proxy"
Oct 30 22:27:46.223639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.223518    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy"
Oct 30 22:27:46.229620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.229588    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.279779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.279742    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:46.286471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.286439    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"]
Oct 30 22:27:46.326906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.326868    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:27:46.344938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.344900    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:46.345313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.345274    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:46.358177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.358139    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.358504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.358468    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.358604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.358587    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzcl\" (UniqueName: \"kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.460313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.460274    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.460523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.460344    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzcl\" (UniqueName: \"kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.460523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.460505    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.461016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.460994    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.463074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.463045    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.519629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.519546    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzcl\" (UniqueName: \"kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl\") pod \"maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.545718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.545667    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:46.934296 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:27:46.934250    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045772a3_9f2a_4649_8645_1c19dc5c761b.slice/crio-ea44c91cefb1a4047840746fbe7943fe6047dd888e2798d29ccd1277073588ea WatchSource:0}: Error finding container ea44c91cefb1a4047840746fbe7943fe6047dd888e2798d29ccd1277073588ea: Status 404 returned error can't find the container with id ea44c91cefb1a4047840746fbe7943fe6047dd888e2798d29ccd1277073588ea
Oct 30 22:27:46.935395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:46.935362    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"]
Oct 30 22:27:47.853357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:47.853309    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57" event={"ID":"045772a3-9f2a-4649-8645-1c19dc5c761b","Type":"ContainerStarted","Data":"63e92f8880447ef2cb9b0932c5c0eea02aecea1e05faf5fcb61875b5ad70c7d9"}
Oct 30 22:27:47.853890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:47.853372    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57" event={"ID":"045772a3-9f2a-4649-8645-1c19dc5c761b","Type":"ContainerStarted","Data":"ea44c91cefb1a4047840746fbe7943fe6047dd888e2798d29ccd1277073588ea"}
Oct 30 22:27:47.969726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:47.969631    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57" podStartSLOduration=1.969617674 podStartE2EDuration="1.969617674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:47.964071885 +0000 UTC m=+1598.813584859" watchObservedRunningTime="2026-10-30 22:27:47.969617674 +0000 UTC m=+1598.819130646"
Oct 30 22:27:48.864594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:48.864550    5028 generic.go:356] "Generic (PLEG): container finished" podID="045772a3-9f2a-4649-8645-1c19dc5c761b" containerID="63e92f8880447ef2cb9b0932c5c0eea02aecea1e05faf5fcb61875b5ad70c7d9" exitCode=0
Oct 30 22:27:48.865181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:48.864630    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57" event={"ID":"045772a3-9f2a-4649-8645-1c19dc5c761b","Type":"ContainerDied","Data":"63e92f8880447ef2cb9b0932c5c0eea02aecea1e05faf5fcb61875b5ad70c7d9"}
Oct 30 22:27:50.159927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.159898    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:50.260993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.260952    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl\" (UniqueName: \"kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl\") pod \"045772a3-9f2a-4649-8645-1c19dc5c761b\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") "
Oct 30 22:27:50.261202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.261052    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca\") pod \"045772a3-9f2a-4649-8645-1c19dc5c761b\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") "
Oct 30 22:27:50.261202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.261086    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca\") pod \"045772a3-9f2a-4649-8645-1c19dc5c761b\" (UID: \"045772a3-9f2a-4649-8645-1c19dc5c761b\") "
Oct 30 22:27:50.261493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.261462    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca" pod "045772a3-9f2a-4649-8645-1c19dc5c761b" (UID: "045772a3-9f2a-4649-8645-1c19dc5c761b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:50.261825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.261806    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.263173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.263152    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca" pod "045772a3-9f2a-4649-8645-1c19dc5c761b" (UID: "045772a3-9f2a-4649-8645-1c19dc5c761b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:50.263352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.263329    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl" pod "045772a3-9f2a-4649-8645-1c19dc5c761b" (UID: "045772a3-9f2a-4649-8645-1c19dc5c761b"). InnerVolumeSpecName "kube-api-access-xzzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:50.363323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.363286    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzcl\" (UniqueName: \"kubernetes.io/projected/045772a3-9f2a-4649-8645-1c19dc5c761b-kube-api-access-xzzcl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.363323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.363324    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045772a3-9f2a-4649-8645-1c19dc5c761b-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.602264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.602182    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:50.755962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.755922    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:50.930150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.930061    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:50.936703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.936662    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57" event={"ID":"045772a3-9f2a-4649-8645-1c19dc5c761b","Type":"ContainerDied","Data":"ea44c91cefb1a4047840746fbe7943fe6047dd888e2798d29ccd1277073588ea"}
Oct 30 22:27:50.936870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.936730    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea44c91cefb1a4047840746fbe7943fe6047dd888e2798d29ccd1277073588ea"
Oct 30 22:27:50.936870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:50.936677    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"
Oct 30 22:27:55.057980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.057943    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"]
Oct 30 22:27:55.103552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.103503    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba7014e-d26zb"]
Oct 30 22:27:55.109737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.109665    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"]
Oct 30 22:27:55.114883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.114857    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5ccb7f-jx79m"]
Oct 30 22:27:55.294764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.294673    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4051c196-a060-49aa-a286-90208a51848a" path="/var/lib/kubelet/pods/4051c196-a060-49aa-a286-90208a51848a/volumes"
Oct 30 22:27:55.295325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.295307    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d5dcbe-e6e7-44ec-abe8-b2141132666a" path="/var/lib/kubelet/pods/66d5dcbe-e6e7-44ec-abe8-b2141132666a/volumes"
Oct 30 22:27:55.860639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:55.860598    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.178010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.177929    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"]
Oct 30 22:27:56.178764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.178747    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy"
Oct 30 22:27:56.178818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.178766    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e660d4aa-1866-4615-a66b-b5d3d499d00b" containerName="istio-proxy"
Oct 30 22:27:56.178818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.178792    5028 cpu_manager.go:414] "containerMap: removing container" podUID="045772a3-9f2a-4649-8645-1c19dc5c761b" containerName="revoke-keys"
Oct 30 22:27:56.178818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.178797    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045772a3-9f2a-4649-8645-1c19dc5c761b" containerName="revoke-keys"
Oct 30 22:27:56.178922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.178913    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="045772a3-9f2a-4649-8645-1c19dc5c761b" containerName="revoke-keys"
Oct 30 22:27:56.184105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.184086    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.241170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.241132    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"]
Oct 30 22:27:56.268896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.268858    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.269096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.269025    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zqf\" (UniqueName: \"kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.269096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.269052    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.282158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.282123    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:56.282359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.282169    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:56.282359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.282212    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:56.282359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.282268    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:27:56.370796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.370759    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.370984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.370899    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zqf\" (UniqueName: \"kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.370984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.370921    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.371494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.371470    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.373393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.373371    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.429990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.429911    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zqf\" (UniqueName: \"kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf\") pod \"maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.499175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.499139    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:27:56.874884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.874845    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.883794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.883764    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"]
Oct 30 22:27:56.999402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:56.999366    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl" event={"ID":"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835","Type":"ContainerStarted","Data":"3342d7cb7b61c59dfb6e0cdd7db5c8a5d0c557de323c815bc45db65d17e491e3"}
Oct 30 22:27:58.012469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:58.012434    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl" event={"ID":"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835","Type":"ContainerStarted","Data":"3a180592566367e5b460970a68153a8fa9b81af80461641729d830490fa84377"}
Oct 30 22:27:58.140533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:58.140474    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl" podStartSLOduration=2.140458146 podStartE2EDuration="2.140458146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:58.124075023 +0000 UTC m=+1608.973587998" watchObservedRunningTime="2026-10-30 22:27:58.140458146 +0000 UTC m=+1608.989971120"
Oct 30 22:27:58.176212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:58.176180    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:58.370863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:58.342313    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:58.463771 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:27:58.463681    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:27:59.024626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:59.024586    5028 generic.go:356] "Generic (PLEG): container finished" podID="ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" containerID="3a180592566367e5b460970a68153a8fa9b81af80461641729d830490fa84377" exitCode=0
Oct 30 22:27:59.025085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:59.024655    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl" event={"ID":"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835","Type":"ContainerDied","Data":"3a180592566367e5b460970a68153a8fa9b81af80461641729d830490fa84377"}
Oct 30 22:27:59.050830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:59.050799    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"]
Oct 30 22:27:59.098056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:59.098019    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77334a6-2bs7t"]
Oct 30 22:27:59.295513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:27:59.295434    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f9b0cc-326e-429d-9a32-1694cafc92a8" path="/var/lib/kubelet/pods/d0f9b0cc-326e-429d-9a32-1694cafc92a8/volumes"
Oct 30 22:28:00.218968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.218942    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:28:00.365504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.365413    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf\" (UniqueName: \"kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf\") pod \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") "
Oct 30 22:28:00.365504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.365466    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca\") pod \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") "
Oct 30 22:28:00.365786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.365765    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca\") pod \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\" (UID: \"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835\") "
Oct 30 22:28:00.366151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.366117    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca" pod "ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" (UID: "ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:00.367053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.367028    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:00.368398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.368361    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf" pod "ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" (UID: "ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835"). InnerVolumeSpecName "kube-api-access-x4zqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:00.369361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.369334    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca" pod "ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" (UID: "ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:00.469321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.469280    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4zqf\" (UniqueName: \"kubernetes.io/projected/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-kube-api-access-x4zqf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:00.469321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:00.469314    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:01.046566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.046529    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"
Oct 30 22:28:01.046770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.046569    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl" event={"ID":"ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835","Type":"ContainerDied","Data":"3342d7cb7b61c59dfb6e0cdd7db5c8a5d0c557de323c815bc45db65d17e491e3"}
Oct 30 22:28:01.046770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.046653    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3342d7cb7b61c59dfb6e0cdd7db5c8a5d0c557de323c815bc45db65d17e491e3"
Oct 30 22:28:01.053352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.053308    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"]
Oct 30 22:28:01.059002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.058973    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"]
Oct 30 22:28:01.094624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.094581    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:01.100306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.100275    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f3b5eaf-sjnkz"]
Oct 30 22:28:01.105774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.105719    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0195b687-nfnrl"]
Oct 30 22:28:01.293798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.293764    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11bc7ef2-5e28-4e1a-89b7-d607394c017e" path="/var/lib/kubelet/pods/11bc7ef2-5e28-4e1a-89b7-d607394c017e/volumes"
Oct 30 22:28:01.294330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:01.294315    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2f8add-e6c4-41c2-aa95-fae5695b1e61" path="/var/lib/kubelet/pods/3b2f8add-e6c4-41c2-aa95-fae5695b1e61/volumes"
Oct 30 22:28:04.064544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:04.064467    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"]
Oct 30 22:28:04.099166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:04.099133    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8cc049-nwtl8"]
Oct 30 22:28:05.295184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:05.295150    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe55cdf-9352-492f-8305-e60795b5c811" path="/var/lib/kubelet/pods/2fe55cdf-9352-492f-8305-e60795b5c811/volumes"
Oct 30 22:28:06.053841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:06.053806    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"]
Oct 30 22:28:06.100537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:06.100496    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c67834-fhsdp"]
Oct 30 22:28:06.189724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:06.189651    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:07.295634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:07.295593    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccc4a53-4ebc-41be-990e-2e279f496a32" path="/var/lib/kubelet/pods/5ccc4a53-4ebc-41be-990e-2e279f496a32/volumes"
Oct 30 22:28:07.332087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:07.332041    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"]
Oct 30 22:28:07.454313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:07.454273    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"]
Oct 30 22:28:07.722198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:07.722157    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:07.911217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:07.911183    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:08.689199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:08.689161    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:09.116720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:09.115019    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:09.263967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:09.263931    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:09.646418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:09.646382    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.066527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.066494    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"]
Oct 30 22:28:10.109058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.109005    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.110568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.110543    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398dc751-6grwc"]
Oct 30 22:28:10.855708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.840321    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"]
Oct 30 22:28:10.857317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.857282    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" containerName="revoke-keys"
Oct 30 22:28:10.857537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.857524    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" containerName="revoke-keys"
Oct 30 22:28:10.857883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.857867    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" containerName="revoke-keys"
Oct 30 22:28:10.893713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.891598    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.903999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.903940    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904031    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904088    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904120    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904151    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904193    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904229    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904308    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zp8j\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.904634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.904446    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:10.974300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.974259    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.978947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.978914    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"]
Oct 30 22:28:10.989237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:10.989169    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5c2f68-openshift-default-dockercfg-5bnch\""
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.037450    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038019    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038079    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038110    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038142    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038183    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038220    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038309    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zp8j\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.038462    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.043788    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.044354    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.045204    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.045452    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.050771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.045703    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.101366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.101328    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.102245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.102221    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.104826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.104781    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.106974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.106919    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zp8j\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j\") pod \"e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") " pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.209318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.209276    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.277037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.276990    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:11.318921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.318844    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e06d734-1ede-46fe-82a0-4f6b7820f9b6" path="/var/lib/kubelet/pods/4e06d734-1ede-46fe-82a0-4f6b7820f9b6/volumes"
Oct 30 22:28:11.509864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.509821    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.670047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.669998    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.846769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.829075    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.947178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.943551    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:11.947178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.943748    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:11.947178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.943860    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:11.978741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:11.978700    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"]
Oct 30 22:28:12.279046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.277569    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" event={"ID":"b1b27184-4cbd-46d1-ade8-c061d5da46a4","Type":"ContainerStarted","Data":"4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"}
Oct 30 22:28:12.279046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.277633    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" event={"ID":"b1b27184-4cbd-46d1-ade8-c061d5da46a4","Type":"ContainerStarted","Data":"691be7ac2d07cd3213bf6e53a0dac404ede0ba1c2f85822354fe0b76dff38ac1"}
Oct 30 22:28:12.279046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.278958    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:12.333731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.332659    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv" podUID="ece83cc6-b519-468e-a2dd-f33432a3ca34" containerName="payload-processing" containerID="cri-o://14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a" gracePeriod=25
Oct 30 22:28:12.398345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.398279    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podStartSLOduration=2.39825993 podStartE2EDuration="2.39825993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:12.396329643 +0000 UTC m=+1623.245842617" watchObservedRunningTime="2026-10-30 22:28:12.39825993 +0000 UTC m=+1623.247772908"
Oct 30 22:28:12.455101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.454967    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx" podUID="29be3106-bfa3-45ad-bacc-5ae7bac0b29d" containerName="payload-pre-processing" containerID="cri-o://50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565" gracePeriod=25
Oct 30 22:28:12.686787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.686744    5028 scope.go:122] "RemoveContainer" containerID="8ca62d47b1a6e1551662a05b260ca667add9203c9fed9467f24356648babcaa7"
Oct 30 22:28:12.708604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.708577    5028 scope.go:122] "RemoveContainer" containerID="c879e29d8fab48708859f04d8e6e8eb9eeb6eea28d9682fff6e86fbcbd7bee06"
Oct 30 22:28:12.773047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.772793    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:28:12.773743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.773598    5028 scope.go:122] "RemoveContainer" containerID="a431d2b8e2599ab673ca2dcceaa93f20018a5c74a3112975a0b218000b7a8f2e"
Oct 30 22:28:12.783243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.783214    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume\") pod \"ece83cc6-b519-468e-a2dd-f33432a3ca34\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") "
Oct 30 22:28:12.783630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.783606    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume" pod "ece83cc6-b519-468e-a2dd-f33432a3ca34" (UID: "ece83cc6-b519-468e-a2dd-f33432a3ca34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:12.783861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.783839    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm\" (UniqueName: \"kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm\") pod \"ece83cc6-b519-468e-a2dd-f33432a3ca34\" (UID: \"ece83cc6-b519-468e-a2dd-f33432a3ca34\") "
Oct 30 22:28:12.785019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.784993    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece83cc6-b519-468e-a2dd-f33432a3ca34-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:12.787973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.787953    5028 scope.go:122] "RemoveContainer" containerID="6ca527349d8106d384f4db65fe0f6343de4f7d79ac6651a949cdf18a38fcd9f8"
Oct 30 22:28:12.792822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.789341    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm" pod "ece83cc6-b519-468e-a2dd-f33432a3ca34" (UID: "ece83cc6-b519-468e-a2dd-f33432a3ca34"). InnerVolumeSpecName "kube-api-access-64kkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:12.814306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.814161    5028 scope.go:122] "RemoveContainer" containerID="1654d3d79b4cd5362646e44df0f591da4ea412c185cb76afffe3b5caae917938"
Oct 30 22:28:12.858596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.858458    5028 scope.go:122] "RemoveContainer" containerID="11776ebb66309eba4b657097981fdf5744af0f320d3b1784c8cb2cb129d792c8"
Oct 30 22:28:12.880812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.880736    5028 scope.go:122] "RemoveContainer" containerID="179c0fa162f31a840447fb8c6c4f85a3d5e1543771777a5e03943f0dff9a4327"
Oct 30 22:28:12.887517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.887466    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64kkm\" (UniqueName: \"kubernetes.io/projected/ece83cc6-b519-468e-a2dd-f33432a3ca34-kube-api-access-64kkm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:12.898811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.898789    5028 scope.go:122] "RemoveContainer" containerID="ee49c7e2e2482477debdfe4fcdeb611d5a8e93fcefc6290e39266bccaa3d441e"
Oct 30 22:28:12.959289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.959254    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:28:12.988865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.988757    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume\") pod \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") "
Oct 30 22:28:12.989018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.988998    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j\" (UniqueName: \"kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j\") pod \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\" (UID: \"29be3106-bfa3-45ad-bacc-5ae7bac0b29d\") "
Oct 30 22:28:12.989104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.989078    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume" pod "29be3106-bfa3-45ad-bacc-5ae7bac0b29d" (UID: "29be3106-bfa3-45ad-bacc-5ae7bac0b29d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:12.989882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.989850    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:12.991232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:12.991201    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j" pod "29be3106-bfa3-45ad-bacc-5ae7bac0b29d" (UID: "29be3106-bfa3-45ad-bacc-5ae7bac0b29d"). InnerVolumeSpecName "kube-api-access-dkr8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:13.091158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.091130    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkr8j\" (UniqueName: \"kubernetes.io/projected/29be3106-bfa3-45ad-bacc-5ae7bac0b29d-kube-api-access-dkr8j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:13.294431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.294398    5028 generic.go:356] "Generic (PLEG): container finished" podID="29be3106-bfa3-45ad-bacc-5ae7bac0b29d" containerID="50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565" exitCode=0
Oct 30 22:28:13.294898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.294504    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"
Oct 30 22:28:13.296420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.296393    5028 generic.go:356] "Generic (PLEG): container finished" podID="ece83cc6-b519-468e-a2dd-f33432a3ca34" containerID="14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a" exitCode=0
Oct 30 22:28:13.296563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.296505    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"
Oct 30 22:28:13.297144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.297122    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:13.298817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298797    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:13.298935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298833    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:28:13.298935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298864    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx" event={"ID":"29be3106-bfa3-45ad-bacc-5ae7bac0b29d","Type":"ContainerDied","Data":"50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565"}
Oct 30 22:28:13.298935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298902    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx" event={"ID":"29be3106-bfa3-45ad-bacc-5ae7bac0b29d","Type":"ContainerDied","Data":"b4a8dd95dde5e4b88d927f33fbb635c8a049b83170c8354ee41f7730ddb82078"}
Oct 30 22:28:13.298935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298930    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv" event={"ID":"ece83cc6-b519-468e-a2dd-f33432a3ca34","Type":"ContainerDied","Data":"14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a"}
Oct 30 22:28:13.299160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298956    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv" event={"ID":"ece83cc6-b519-468e-a2dd-f33432a3ca34","Type":"ContainerDied","Data":"b4355627675fc61fe7ee538b874fb551d54c17e5a1f3bce5ed62e3b7b4707546"}
Oct 30 22:28:13.299160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.298989    5028 scope.go:122] "RemoveContainer" containerID="50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565"
Oct 30 22:28:13.314190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.314171    5028 scope.go:122] "RemoveContainer" containerID="50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565"
Oct 30 22:28:13.314510 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:28:13.314485    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565\": container with ID starting with 50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565 not found: ID does not exist" containerID="50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565"
Oct 30 22:28:13.314560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.314523    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565"} err="failed to get container status \"50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565\": rpc error: code = NotFound desc = could not find container \"50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565\": container with ID starting with 50857c3321a16cacd85df0615b4f6bba637f555e8df0a11ce63f5e12e5737565 not found: ID does not exist"
Oct 30 22:28:13.314560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.314549    5028 scope.go:122] "RemoveContainer" containerID="14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a"
Oct 30 22:28:13.410296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.410270    5028 scope.go:122] "RemoveContainer" containerID="14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a"
Oct 30 22:28:13.410951 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:28:13.410928    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a\": container with ID starting with 14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a not found: ID does not exist" containerID="14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a"
Oct 30 22:28:13.411059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.410963    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a"} err="failed to get container status \"14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a\": rpc error: code = NotFound desc = could not find container \"14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a\": container with ID starting with 14effcd044d1869a3ab1504d0e88dedc923f71a6482ba89396a4e8dc3fd3449a not found: ID does not exist"
Oct 30 22:28:13.465877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.465785    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"]
Oct 30 22:28:13.512206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.512169    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e8ed1149-78cd6657f5-ph6zx"]
Oct 30 22:28:13.604571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.604524    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"]
Oct 30 22:28:13.611385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:13.611351    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e8ed1149-755778c777-nnmqv"]
Oct 30 22:28:15.296682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:15.296644    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29be3106-bfa3-45ad-bacc-5ae7bac0b29d" path="/var/lib/kubelet/pods/29be3106-bfa3-45ad-bacc-5ae7bac0b29d/volumes"
Oct 30 22:28:15.297301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:15.297280    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece83cc6-b519-468e-a2dd-f33432a3ca34" path="/var/lib/kubelet/pods/ece83cc6-b519-468e-a2dd-f33432a3ca34/volumes"
Oct 30 22:28:15.303047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:15.302992    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.472567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:15.472533    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.748720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:15.745610    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:16.056792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:16.056698    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"]
Oct 30 22:28:16.062236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:16.062204    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8926cb-2jm6b"]
Oct 30 22:28:16.693534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:16.693497    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:17.295960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:17.295924    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d070047-c20a-445c-b2ed-2d8f138543d0" path="/var/lib/kubelet/pods/4d070047-c20a-445c-b2ed-2d8f138543d0/volumes"
Oct 30 22:28:20.753057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:20.753021    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.985175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:20.985138    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:21.166267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.166180    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"]
Oct 30 22:28:21.167780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.167750    5028 cpu_manager.go:414] "containerMap: removing container" podUID="29be3106-bfa3-45ad-bacc-5ae7bac0b29d" containerName="payload-pre-processing"
Oct 30 22:28:21.167931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.167789    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29be3106-bfa3-45ad-bacc-5ae7bac0b29d" containerName="payload-pre-processing"
Oct 30 22:28:21.167931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.167846    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ece83cc6-b519-468e-a2dd-f33432a3ca34" containerName="payload-processing"
Oct 30 22:28:21.167931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.167855    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece83cc6-b519-468e-a2dd-f33432a3ca34" containerName="payload-processing"
Oct 30 22:28:21.168196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.168110    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece83cc6-b519-468e-a2dd-f33432a3ca34" containerName="payload-processing"
Oct 30 22:28:21.168196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.168149    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="29be3106-bfa3-45ad-bacc-5ae7bac0b29d" containerName="payload-pre-processing"
Oct 30 22:28:21.189776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.189732    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.253047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.253008    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"]
Oct 30 22:28:21.280714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.279768    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5c2f68-dockercfg-pplkj\""
Oct 30 22:28:21.280714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.280559    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5c2f68\""
Oct 30 22:28:21.413815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.411964    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:21.413815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.413177    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxshg\" (UniqueName: \"kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg\") pod \"payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.413815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.413352    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.514647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.514596    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxshg\" (UniqueName: \"kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg\") pod \"payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.514936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.514762    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.521765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.515890    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.547767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.543472    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"]
Oct 30 22:28:21.602709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.602621    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"]
Oct 30 22:28:21.625701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.611166    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.625701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.616566    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rm8\" (UniqueName: \"kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8\") pod \"payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.625701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.616748    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.625701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.620436    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxshg\" (UniqueName: \"kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg\") pod \"payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.721180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.721139    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rm8\" (UniqueName: \"kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8\") pod \"payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.721389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.721250    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.722257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.722228    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.785599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.785506    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rm8\" (UniqueName: \"kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8\") pod \"payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:21.865960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.865913    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:21.995162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:21.995038    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:22.305340 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:28:22.305304    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe445dba_eacb_43e9_9745_9c569c194037.slice/crio-f6df51895eec6a45fe140a19707c387ea8fe3760135b750d8a3603942496d476 WatchSource:0}: Error finding container f6df51895eec6a45fe140a19707c387ea8fe3760135b750d8a3603942496d476: Status 404 returned error can't find the container with id f6df51895eec6a45fe140a19707c387ea8fe3760135b750d8a3603942496d476
Oct 30 22:28:22.305994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.305897    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"]
Oct 30 22:28:22.447937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.444902    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"]
Oct 30 22:28:22.595247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.595213    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc" event={"ID":"e3f74408-b90d-4335-ac65-afe19b9bf935","Type":"ContainerStarted","Data":"74f332a4c55a0c89f72493ee26472cc899a512187de5092995592dc5b74e6041"}
Oct 30 22:28:22.597112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.597083    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" event={"ID":"fe445dba-eacb-43e9-9745-9c569c194037","Type":"ContainerStarted","Data":"2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f"}
Oct 30 22:28:22.597255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.597142    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:22.597255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.597165    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" event={"ID":"fe445dba-eacb-43e9-9745-9c569c194037","Type":"ContainerStarted","Data":"f6df51895eec6a45fe140a19707c387ea8fe3760135b750d8a3603942496d476"}
Oct 30 22:28:22.708701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:22.708641    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" podStartSLOduration=1.7086268420000001 podStartE2EDuration="1.708626842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:22.708031208 +0000 UTC m=+1633.557544181" watchObservedRunningTime="2026-10-30 22:28:22.708626842 +0000 UTC m=+1633.558139815"
Oct 30 22:28:23.162869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.162816    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:28:23.163555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.162876    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:23.275013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.274977    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"]
Oct 30 22:28:23.275236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.275211    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" containerID="cri-o://7524e44f8975ec5517db7f8ea26de596e269c37337d018ffa586e816f8144a31" gracePeriod=30
Oct 30 22:28:23.312779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.312742    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"]
Oct 30 22:28:23.312990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.312966    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy" containerID="cri-o://3e4f64f74329ce9f3392af8e44ef7c4b5c8f1cf1cf835f7484b07654b87e8e11" gracePeriod=30
Oct 30 22:28:23.322457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.322423    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"]
Oct 30 22:28:23.322661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.322623    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy" containerID="cri-o://1c1619c85c55dd42a4aa3a0e90722d9d61b5038de54ff1537f1848bd7a4b4ba0" gracePeriod=30
Oct 30 22:28:23.612909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.612873    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc" event={"ID":"e3f74408-b90d-4335-ac65-afe19b9bf935","Type":"ContainerStarted","Data":"2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889"}
Oct 30 22:28:23.678947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.612980    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:23.777938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:23.777889    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc" podStartSLOduration=2.777872736 podStartE2EDuration="2.777872736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.776234429 +0000 UTC m=+1634.625747402" watchObservedRunningTime="2026-10-30 22:28:23.777872736 +0000 UTC m=+1634.627385759"
Oct 30 22:28:24.624808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:24.624771    5028 generic.go:356] "Generic (PLEG): container finished" podID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerID="8c1b43780b2afafc3df4d78b7fd05bc8ca91bb2e49bf3f9b25dd8ec79cfbe3d0" exitCode=0
Oct 30 22:28:24.625316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:24.624850    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" event={"ID":"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5","Type":"ContainerDied","Data":"8c1b43780b2afafc3df4d78b7fd05bc8ca91bb2e49bf3f9b25dd8ec79cfbe3d0"}
Oct 30 22:28:24.625448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:24.625409    5028 scope.go:122] "RemoveContainer" containerID="8c1b43780b2afafc3df4d78b7fd05bc8ca91bb2e49bf3f9b25dd8ec79cfbe3d0"
Oct 30 22:28:24.627454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:24.627421    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:24.627555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:24.627511    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:24.627593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:24.627559    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:25.668112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:25.668066    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" event={"ID":"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5","Type":"ContainerStarted","Data":"cc275c72232d920e36051b3cfc878671cbfb9fbe8017ef892ac540ae8f853aac"}
Oct 30 22:28:25.983852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:25.983812    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:26.107200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:26.107156    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:28:26.107435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:26.107408    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:28:26.107556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:26.107468    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 22:28:27.057532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.057485    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"]
Oct 30 22:28:27.101598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.101563    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f705abe-hb7rv"]
Oct 30 22:28:27.107629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.107599    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:28:27.107810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.107656    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 22:28:27.201660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.201623    5028 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:28:27.201855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.201679    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:28:27.293796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:27.293763    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fc6480-f647-45a4-a1da-8263220c3a5c" path="/var/lib/kubelet/pods/f7fc6480-f647-45a4-a1da-8263220c3a5c/volumes"
Oct 30 22:28:28.108058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.108017    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:28:28.108483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.108080    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 22:28:28.486080 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:28:28.486045    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:28:28.711514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.711474    5028 generic.go:356] "Generic (PLEG): container finished" podID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerID="7524e44f8975ec5517db7f8ea26de596e269c37337d018ffa586e816f8144a31" exitCode=0
Oct 30 22:28:28.711771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.711655    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" event={"ID":"b2ac1dbd-e936-4d43-8086-1dd727ada958","Type":"ContainerDied","Data":"7524e44f8975ec5517db7f8ea26de596e269c37337d018ffa586e816f8144a31"}
Oct 30 22:28:28.711771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.711726    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9" event={"ID":"b2ac1dbd-e936-4d43-8086-1dd727ada958","Type":"ContainerDied","Data":"2cfdbc611ca6160b128a191e4e59c4963e5112f94c945a7adb64cd6269078f93"}
Oct 30 22:28:28.711771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.711751    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cfdbc611ca6160b128a191e4e59c4963e5112f94c945a7adb64cd6269078f93"
Oct 30 22:28:28.714256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.714093    5028 generic.go:356] "Generic (PLEG): container finished" podID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerID="3e4f64f74329ce9f3392af8e44ef7c4b5c8f1cf1cf835f7484b07654b87e8e11" exitCode=0
Oct 30 22:28:28.714256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.714169    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" event={"ID":"2f1933fb-6dac-4791-89cc-04fabc2aaaa3","Type":"ContainerDied","Data":"3e4f64f74329ce9f3392af8e44ef7c4b5c8f1cf1cf835f7484b07654b87e8e11"}
Oct 30 22:28:28.746828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.746790    5028 generic.go:356] "Generic (PLEG): container finished" podID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerID="1c1619c85c55dd42a4aa3a0e90722d9d61b5038de54ff1537f1848bd7a4b4ba0" exitCode=0
Oct 30 22:28:28.747102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.747075    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" event={"ID":"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a","Type":"ContainerDied","Data":"1c1619c85c55dd42a4aa3a0e90722d9d61b5038de54ff1537f1848bd7a4b4ba0"}
Oct 30 22:28:28.753998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.753915    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:28:28.773457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.773420    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:28:28.845843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.845801    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:28:28.886638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886599    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.886880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886646    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.886880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886721    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.886880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886799    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.886880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886852    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886892    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886919    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886951    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886996    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.886981    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887028    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887050    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887105    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.887515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887138    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.887515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887197    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.887515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887257    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887341    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887383    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887411    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.887835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887511    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:28.887835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887590    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l\") pod \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\" (UID: \"2f1933fb-6dac-4791-89cc-04fabc2aaaa3\") "
Oct 30 22:28:28.887835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887637    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy\") pod \"b2ac1dbd-e936-4d43-8086-1dd727ada958\" (UID: \"b2ac1dbd-e936-4d43-8086-1dd727ada958\") "
Oct 30 22:28:28.887987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.887909    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:28.888822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888514    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.888822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888554    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.888822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888627    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.888822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888757    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.888822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888779    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888915    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888934    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888950    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888965    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ac1dbd-e936-4d43-8086-1dd727ada958-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888980    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.888995    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.889009    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.889796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.889712    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:28.890319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.890297    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.890423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.890333    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.890423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.890398    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:28.890498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.890431    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "kube-api-access-2vlw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:28.891082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.891059    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "kube-api-access-q7g6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:28.891180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.891111    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token" pod "b2ac1dbd-e936-4d43-8086-1dd727ada958" (UID: "b2ac1dbd-e936-4d43-8086-1dd727ada958"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:28.891802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.891780    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo" pod "2f1933fb-6dac-4791-89cc-04fabc2aaaa3" (UID: "2f1933fb-6dac-4791-89cc-04fabc2aaaa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:28.989821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.989779    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.989871    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.989919    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990006    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990052    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990097    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990094    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.990215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990178    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990302    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990345    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data\") pod \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\" (UID: \"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a\") "
Oct 30 22:28:28.990520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990375    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.990779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990638    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:28.990852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.990833    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.991990    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992017    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992031    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992057    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992071    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992087    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992100    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992113    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992127    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7g6l\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-kube-api-access-q7g6l\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992141    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992154    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992166    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992179    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992191    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vlw6\" (UniqueName: \"kubernetes.io/projected/b2ac1dbd-e936-4d43-8086-1dd727ada958-kube-api-access-2vlw6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992206    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1933fb-6dac-4791-89cc-04fabc2aaaa3-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:28.992296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.992268    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.993369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.993338    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:28.993369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.993342    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:28.993540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.993408    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "kube-api-access-6wqrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:28.993540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:28.993506    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token" pod "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" (UID: "8c7871f4-4a43-42cd-ae4a-33e3f89cd26a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:29.094569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.094529    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.094569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.094572    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.094829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.094588    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.094829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.094602    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqrg\" (UniqueName: \"kubernetes.io/projected/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-kube-api-access-6wqrg\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.094829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.094617    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.108172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.108132    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:28:29.108668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.108196    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 22:28:29.773310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.773276    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"
Oct 30 22:28:29.773953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.773926    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt" event={"ID":"2f1933fb-6dac-4791-89cc-04fabc2aaaa3","Type":"ContainerDied","Data":"49ad1b05552fd82d54eb369e3a9a0660b2956671f79a1a59e929e824922a63b5"}
Oct 30 22:28:29.774090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.774019    5028 scope.go:122] "RemoveContainer" containerID="3e4f64f74329ce9f3392af8e44ef7c4b5c8f1cf1cf835f7484b07654b87e8e11"
Oct 30 22:28:29.776181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.776157    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"
Oct 30 22:28:29.777016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.776978    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj" event={"ID":"8c7871f4-4a43-42cd-ae4a-33e3f89cd26a","Type":"ContainerDied","Data":"0f20611e6db03448948435323c85dedbfd3c97934dffce80f8cac4f6a2dc63f8"}
Oct 30 22:28:29.777133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.776997    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"
Oct 30 22:28:29.789356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:29.789335    5028 scope.go:122] "RemoveContainer" containerID="1c1619c85c55dd42a4aa3a0e90722d9d61b5038de54ff1537f1848bd7a4b4ba0"
Oct 30 22:28:30.014715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.010461    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"]
Oct 30 22:28:30.029736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.029624    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-7xttt"]
Oct 30 22:28:30.107824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.107786    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:28:30.108003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.107857    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 22:28:30.170994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.170923    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"]
Oct 30 22:28:30.185044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.184972    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-kmnzj"]
Oct 30 22:28:30.270492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.270455    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"]
Oct 30 22:28:30.278275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.278234    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-pnvl9"]
Oct 30 22:28:30.781912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:30.781878    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:31.054053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.053955    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"]
Oct 30 22:28:31.058951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.058920    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3edaca53-nt87t"]
Oct 30 22:28:31.117462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.117412    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:28:31.118073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.118044    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:28:31.119079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.119057    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:28:31.294477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.294446    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" path="/var/lib/kubelet/pods/2f1933fb-6dac-4791-89cc-04fabc2aaaa3/volumes"
Oct 30 22:28:31.295081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.295066    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" path="/var/lib/kubelet/pods/8c7871f4-4a43-42cd-ae4a-33e3f89cd26a/volumes"
Oct 30 22:28:31.295581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.295560    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" path="/var/lib/kubelet/pods/b2ac1dbd-e936-4d43-8086-1dd727ada958/volumes"
Oct 30 22:28:31.296599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:31.296581    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd" path="/var/lib/kubelet/pods/dd69d1f0-6d5f-49cd-9f3f-0c12cf2257dd/volumes"
Oct 30 22:28:33.616705 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:33.616662    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:28:34.102954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:34.102917    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:34.576136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:34.576086    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:34.628570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:34.628539    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.280805    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"]
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282043    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282064    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282131    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282140    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282159    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282168    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282333    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1933fb-6dac-4791-89cc-04fabc2aaaa3" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282354    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ac1dbd-e936-4d43-8086-1dd727ada958" containerName="istio-proxy"
Oct 30 22:28:35.284745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.282363    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7871f4-4a43-42cd-ae4a-33e3f89cd26a" containerName="istio-proxy"
Oct 30 22:28:35.304420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.298995    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.418496    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-bbd10f-openshift-default-dockercfg-76rd7\""
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420025    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6wb\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420081    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420128    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420227    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420366    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420450    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420570    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420597    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.420624    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.427979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.423385    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:35.523211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.523164    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.523594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.523571    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.523732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.523710    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.523876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.523861    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.524009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.523995    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6wb\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.524117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.524105    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.524233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.524221    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.524420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.524408    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.524645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.524633    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.546627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.534782    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.546627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.535384    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.546627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.535631    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.546627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.536152    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.546627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.536391    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.546627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.540349    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.554788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.552353    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.567052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.566433    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"]
Oct 30 22:28:35.593906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.593870    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.594074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.593968    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6wb\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb\") pod \"e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") " pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.753892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.753861    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:35.811450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.811177    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:35.911221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:35.910399    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:36.271089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.271054    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"]
Oct 30 22:28:36.298998 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:28:36.298967    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6de087b_d027_4356_a2a2_4e55b5a258cf.slice/crio-354d954016a34ec2b33ace6c5fa6ea2e689bedce6b259f9573ccbd6e638a9ce5 WatchSource:0}: Error finding container 354d954016a34ec2b33ace6c5fa6ea2e689bedce6b259f9573ccbd6e638a9ce5: Status 404 returned error can't find the container with id 354d954016a34ec2b33ace6c5fa6ea2e689bedce6b259f9573ccbd6e638a9ce5
Oct 30 22:28:36.301886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.301852    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:36.301977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.301928    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:36.301977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.301962    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:36.861675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.861636    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" event={"ID":"f6de087b-d027-4356-a2a2-4e55b5a258cf","Type":"ContainerStarted","Data":"7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"}
Oct 30 22:28:36.862216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.861705    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" event={"ID":"f6de087b-d027-4356-a2a2-4e55b5a258cf","Type":"ContainerStarted","Data":"354d954016a34ec2b33ace6c5fa6ea2e689bedce6b259f9573ccbd6e638a9ce5"}
Oct 30 22:28:36.979731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:36.979659    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podStartSLOduration=1.979643291 podStartE2EDuration="1.979643291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:36.977106985 +0000 UTC m=+1647.826619975" watchObservedRunningTime="2026-10-30 22:28:36.979643291 +0000 UTC m=+1647.829156264"
Oct 30 22:28:37.812027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:37.811993    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:37.817925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:37.817889    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:37.871906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:37.871874    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:37.872929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:37.872908    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:28:38.540618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:38.540584    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:38.698138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:38.698101    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:39.785078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:39.785048    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:39.961006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:39.960965    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:40.112026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:40.111902    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:40.686036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:40.685994    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:42.070976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:42.070937    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"]
Oct 30 22:28:42.118107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:42.118065    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d746664c-v8fnz"]
Oct 30 22:28:42.944891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:42.944852    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:43.102014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.101948    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:43.204736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.204598    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"]
Oct 30 22:28:43.209668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.209639    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0253345-m8ddq"]
Oct 30 22:28:43.294444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.294406    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d568fd-de6c-43d7-bfe5-6d7137646a54" path="/var/lib/kubelet/pods/19d568fd-de6c-43d7-bfe5-6d7137646a54/volumes"
Oct 30 22:28:43.295230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.295200    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b725fd6a-7e97-45ab-baec-a5a5c35970cf" path="/var/lib/kubelet/pods/b725fd6a-7e97-45ab-baec-a5a5c35970cf/volumes"
Oct 30 22:28:43.329164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.329128    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"]
Oct 30 22:28:43.335619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.335592    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.386372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.386340    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-bbd10f\""
Oct 30 22:28:43.386538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.386470    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-bbd10f-dockercfg-b54qp\""
Oct 30 22:28:43.391194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.391163    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"]
Oct 30 22:28:43.400669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.400632    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.401203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.401171    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grz8m\" (UniqueName: \"kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m\") pod \"payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.537100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.536867    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grz8m\" (UniqueName: \"kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m\") pod \"payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.537565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.537538    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.538747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.538620    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.584044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.583622    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"]
Oct 30 22:28:43.596749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.593069    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.710290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.710256    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"]
Oct 30 22:28:43.724712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.712985    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grz8m\" (UniqueName: \"kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m\") pod \"payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:43.743719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.743495    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vx47\" (UniqueName: \"kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47\") pod \"payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.743897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.743818    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.847926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.847810    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.848300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.847998    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vx47\" (UniqueName: \"kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47\") pod \"payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.849303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.849270    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.908248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.908216    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vx47\" (UniqueName: \"kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47\") pod \"payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:43.959945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:43.959901    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:44.068169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.063975    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:44.353739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.353161    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"]
Oct 30 22:28:44.354282 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:28:44.354251    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e9157c_68a6_4bde_a2c2_8782222c725b.slice/crio-b506a6535a3608c89e3d608b38740d799812deef3c86e73c9aa7ee7084138978 WatchSource:0}: Error finding container b506a6535a3608c89e3d608b38740d799812deef3c86e73c9aa7ee7084138978: Status 404 returned error can't find the container with id b506a6535a3608c89e3d608b38740d799812deef3c86e73c9aa7ee7084138978
Oct 30 22:28:44.485756 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:28:44.485716    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc6a178_d6b4_4a68_ac89_12013e30d113.slice/crio-018d172e94d553bde3b14e0f93f47b3b7dbe2b0f912367a65a81996842fdf059 WatchSource:0}: Error finding container 018d172e94d553bde3b14e0f93f47b3b7dbe2b0f912367a65a81996842fdf059: Status 404 returned error can't find the container with id 018d172e94d553bde3b14e0f93f47b3b7dbe2b0f912367a65a81996842fdf059
Oct 30 22:28:44.485995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.485972    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"]
Oct 30 22:28:44.954657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.954613    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l" event={"ID":"9cc6a178-d6b4-4a68-ac89-12013e30d113","Type":"ContainerStarted","Data":"3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e"}
Oct 30 22:28:44.954878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.954724    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l" event={"ID":"9cc6a178-d6b4-4a68-ac89-12013e30d113","Type":"ContainerStarted","Data":"018d172e94d553bde3b14e0f93f47b3b7dbe2b0f912367a65a81996842fdf059"}
Oct 30 22:28:44.955824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.955790    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:44.957922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.957897    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6" event={"ID":"94e9157c-68a6-4bde-a2c2-8782222c725b","Type":"ContainerStarted","Data":"2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5"}
Oct 30 22:28:44.958045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.957946    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6" event={"ID":"94e9157c-68a6-4bde-a2c2-8782222c725b","Type":"ContainerStarted","Data":"b506a6535a3608c89e3d608b38740d799812deef3c86e73c9aa7ee7084138978"}
Oct 30 22:28:44.958045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:44.957985    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:45.070078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:45.070023    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l" podStartSLOduration=2.070009522 podStartE2EDuration="2.070009522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:45.067797594 +0000 UTC m=+1655.917310570" watchObservedRunningTime="2026-10-30 22:28:45.070009522 +0000 UTC m=+1655.919522489"
Oct 30 22:28:45.180609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:45.180537    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6" podStartSLOduration=2.180518035 podStartE2EDuration="2.180518035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:45.179792629 +0000 UTC m=+1656.029305614" watchObservedRunningTime="2026-10-30 22:28:45.180518035 +0000 UTC m=+1656.030031009"
Oct 30 22:28:45.776708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:45.776553    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:46.062771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:46.060310    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"]
Oct 30 22:28:46.105359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:46.105328    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d1d18a5-cn92n"]
Oct 30 22:28:47.295753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:47.295722    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1ae48e-31b2-48dc-a3a6-f00721ad662c" path="/var/lib/kubelet/pods/ee1ae48e-31b2-48dc-a3a6-f00721ad662c/volumes"
Oct 30 22:28:47.401831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:47.401797    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.770955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:47.770918    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.040234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:48.040145    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:48.361470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:48.361376    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"]
Oct 30 22:28:48.407984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:48.407949    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc8c7d75-5zw57"]
Oct 30 22:28:48.448908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:48.448869    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:49.295115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.295039    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045772a3-9f2a-4649-8645-1c19dc5c761b" path="/var/lib/kubelet/pods/045772a3-9f2a-4649-8645-1c19dc5c761b/volumes"
Oct 30 22:28:49.319868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.319834    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:49.833295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.833255    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"]
Oct 30 22:28:49.841619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.841592    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.869754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.869717    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctq6\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.869929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.869812    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.869929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.869847    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.869929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.869882    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.870132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.869932    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.870132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.869982    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.870132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.870016    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.870294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.870223    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.870382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.870365    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.897587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.895960    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-71c235-openshift-default-dockercfg-zkmb2\""
Oct 30 22:28:49.940721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.940019    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:49.959819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.958515    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"]
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.992711    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.992993    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993038    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993314    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993483    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993545    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctq6\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993620    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993649    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.993701    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.996472    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:49.997711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:49.996892    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.011727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.000047    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.011727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.000266    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.011727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.000448    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.011727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.003487    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.029712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.029128    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.117985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.117831    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctq6\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.118703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.118642    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token\") pod \"e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") " pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.175875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.175589    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:50.281153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.281120    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:50.688264 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:28:50.688228    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5246328d_a9d6_4499_8fd0_c05902265768.slice/crio-640268a8463104a1743d5fe98dbdceb33ea33d44e921695ed6763be65097d2b6 WatchSource:0}: Error finding container 640268a8463104a1743d5fe98dbdceb33ea33d44e921695ed6763be65097d2b6: Status 404 returned error can't find the container with id 640268a8463104a1743d5fe98dbdceb33ea33d44e921695ed6763be65097d2b6
Oct 30 22:28:50.689591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.689565    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"]
Oct 30 22:28:50.691447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.691412    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:50.691522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.691492    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:50.691556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:50.691525    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:51.151130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.151088    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" event={"ID":"5246328d-a9d6-4499-8fd0-c05902265768","Type":"ContainerStarted","Data":"5db3033cf2e5ba970a552d9b29365013c1d6c32e208b523243dc31b6a3cfaedd"}
Oct 30 22:28:51.151636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.151158    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" event={"ID":"5246328d-a9d6-4499-8fd0-c05902265768","Type":"ContainerStarted","Data":"640268a8463104a1743d5fe98dbdceb33ea33d44e921695ed6763be65097d2b6"}
Oct 30 22:28:51.176399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.176360    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:51.178074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.178045    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:28:51.178204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.178111    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:28:51.277175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.277122    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podStartSLOduration=2.277107637 podStartE2EDuration="2.277107637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:51.274111574 +0000 UTC m=+1662.123624548" watchObservedRunningTime="2026-10-30 22:28:51.277107637 +0000 UTC m=+1662.126620609"
Oct 30 22:28:51.858000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:51.857965    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:52.190785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:52.189337    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:53.180676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:53.180640    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:53.181821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:53.181797    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:28:53.519899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:53.519855    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:53.851988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:53.851899    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.008944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.008913    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.052253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.052216    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"]
Oct 30 22:28:54.061560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.061516    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e8ed1149-85wvl"]
Oct 30 22:28:54.170121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.170002    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.446148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.446051    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.684767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.684733    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.861710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:54.858248    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:55.033714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.030526    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:55.295039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.294993    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835" path="/var/lib/kubelet/pods/ae2009ce-1ce9-4e9d-bbdb-4277e5f5e835/volumes"
Oct 30 22:28:55.629768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.629634    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"]
Oct 30 22:28:55.891410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.891308    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:55.961373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.957929    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"]
Oct 30 22:28:55.962314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.962279    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:55.991871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:55.991840    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:28:56.013479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.013442    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-73735c-openshift-default-dockercfg-fjddp\""
Oct 30 22:28:56.032738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.032319    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.033276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.033227    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.035733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.035658    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.035733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.035723    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.035927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.035768    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5l9l\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.036515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.036488    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.036828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.036803    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.037158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.037139    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.037409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.037392    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149029    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149147    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149234    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149306    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149357    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149383    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149423    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5l9l\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149469    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.149546    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.150228    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.151759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.151358    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.154740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.153893    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.154740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.154379    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.157114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.155065    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.157283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.156152    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.157387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.156919    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.163337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.163308    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:56.212710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.212536    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5l9l\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.216714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.215157    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token\") pod \"e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") " pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.289715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.288967    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:56.781145 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:28:56.781113    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed80fe5_2bbd_4988_a7d9_da7026e965c0.slice/crio-126e35175c55030f0dd48201e148a0dca9e2a8277e729c678a7f2f78ab3a13ee WatchSource:0}: Error finding container 126e35175c55030f0dd48201e148a0dca9e2a8277e729c678a7f2f78ab3a13ee: Status 404 returned error can't find the container with id 126e35175c55030f0dd48201e148a0dca9e2a8277e729c678a7f2f78ab3a13ee
Oct 30 22:28:56.781534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.781512    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"]
Oct 30 22:28:56.784260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.784228    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:56.784349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.784293    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:56.784349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.784323    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:56.997519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:56.997489    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:28:57.246750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:57.246715    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" event={"ID":"6ed80fe5-2bbd-4988-a7d9-da7026e965c0","Type":"ContainerStarted","Data":"99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"}
Oct 30 22:28:57.247052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:57.246780    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" event={"ID":"6ed80fe5-2bbd-4988-a7d9-da7026e965c0","Type":"ContainerStarted","Data":"126e35175c55030f0dd48201e148a0dca9e2a8277e729c678a7f2f78ab3a13ee"}
Oct 30 22:28:57.295558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:57.295526    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:57.296951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:57.296917    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:57.404833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:57.404768    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podStartSLOduration=2.404748174 podStartE2EDuration="2.404748174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:57.398954999 +0000 UTC m=+1668.248467971" watchObservedRunningTime="2026-10-30 22:28:57.404748174 +0000 UTC m=+1668.254261151"
Oct 30 22:28:58.256756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:58.256721    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:58.257816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:58.257789    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:28:58.460190 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:28:58.460146    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:28:59.672603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:59.672566    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:59.922756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:28:59.922594    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:00.122594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.118538    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:00.525714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.512283    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"]
Oct 30 22:29:00.549784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.547990    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.573859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.573774    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"]
Oct 30 22:29:00.612926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.609170    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-71c235-dockercfg-dhcxj\""
Oct 30 22:29:00.612926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.609834    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-71c235\""
Oct 30 22:29:00.673391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.673353    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"]
Oct 30 22:29:00.682945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.682027    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.683823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.683784    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nxl\" (UniqueName: \"kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl\") pod \"payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.684031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.684011    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.749923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.749866    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"]
Oct 30 22:29:00.785947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.785840    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nxl\" (UniqueName: \"kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl\") pod \"payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.785947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.785893    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.786198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.786122    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.786285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.786269    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8t8\" (UniqueName: \"kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8\") pod \"payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.787165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.787139    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.847657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.847621    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nxl\" (UniqueName: \"kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl\") pod \"payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.888225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.888187    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:00.889065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.889029    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8t8\" (UniqueName: \"kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8\") pod \"payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.889600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.889571    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.890895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.890873    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.953758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.953447    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8t8\" (UniqueName: \"kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8\") pod \"payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:00.963112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:00.962995    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:01.009726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:01.005143    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:01.126269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:01.126045    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:01.399577 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:29:01.399546    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0715e293_39d3_4b44_8429_53ab9827126e.slice/crio-f1b18e3fa82416528b655525ab29548749959bdc503fe4915e8d4b7b52ed2e23 WatchSource:0}: Error finding container f1b18e3fa82416528b655525ab29548749959bdc503fe4915e8d4b7b52ed2e23: Status 404 returned error can't find the container with id f1b18e3fa82416528b655525ab29548749959bdc503fe4915e8d4b7b52ed2e23
Oct 30 22:29:01.400427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:01.400401    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"]
Oct 30 22:29:01.511749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:01.511717    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"]
Oct 30 22:29:01.512921 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:29:01.512895    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2194013_886a_48da_992e_d00192443e1f.slice/crio-aef0557fa02dd34c079296b182c4128a880d7999307b2389ee2cebd347adbcf8 WatchSource:0}: Error finding container aef0557fa02dd34c079296b182c4128a880d7999307b2389ee2cebd347adbcf8: Status 404 returned error can't find the container with id aef0557fa02dd34c079296b182c4128a880d7999307b2389ee2cebd347adbcf8
Oct 30 22:29:01.565939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:01.565906    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:02.284752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.281402    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"]
Oct 30 22:29:02.314709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.313771    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.341744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.331495    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.347569    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.347635    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.347847    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.348023    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.348548    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlpr\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.350329    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.348609    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.350607    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.350709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.350675    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.464716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.453334    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a3600a-openshift-default-dockercfg-6qk85\""
Oct 30 22:29:02.467397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467352    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.467588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467500    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.467655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467610    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlpr\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.467796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467661    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.467796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467774    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.467910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467834    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.467910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467889    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.468022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467954    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.468022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.467997    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.495804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.490906    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"]
Oct 30 22:29:02.495804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.491277    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.495804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.491323    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.495804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.491528    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.495804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.491591    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.501034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.501001    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.522406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.522370    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.530067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.530043    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.534823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.533468    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn" event={"ID":"c2194013-886a-48da-992e-d00192443e1f","Type":"ContainerStarted","Data":"66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e"}
Oct 30 22:29:02.534823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.533528    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn" event={"ID":"c2194013-886a-48da-992e-d00192443e1f","Type":"ContainerStarted","Data":"aef0557fa02dd34c079296b182c4128a880d7999307b2389ee2cebd347adbcf8"}
Oct 30 22:29:02.534823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.534715    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:02.537756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.537723    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2" event={"ID":"0715e293-39d3-4b44-8429-53ab9827126e","Type":"ContainerStarted","Data":"c6ad0321a6608f7e0319230c3376b29747d85247745575c5866fe61463faca65"}
Oct 30 22:29:02.537869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.537773    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2" event={"ID":"0715e293-39d3-4b44-8429-53ab9827126e","Type":"ContainerStarted","Data":"f1b18e3fa82416528b655525ab29548749959bdc503fe4915e8d4b7b52ed2e23"}
Oct 30 22:29:02.538615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.538593    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:02.590659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.590624    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlpr\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.604372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.604317    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token\") pod \"e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") " pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.667728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.665369    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:02.689740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.687401    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn" podStartSLOduration=2.687382306 podStartE2EDuration="2.687382306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:02.661871534 +0000 UTC m=+1673.511384504" watchObservedRunningTime="2026-10-30 22:29:02.687382306 +0000 UTC m=+1673.536895376"
Oct 30 22:29:02.692935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.692900    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:02.792340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.790109    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2" podStartSLOduration=2.790086854 podStartE2EDuration="2.790086854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:02.781343679 +0000 UTC m=+1673.630856656" watchObservedRunningTime="2026-10-30 22:29:02.790086854 +0000 UTC m=+1673.639599828"
Oct 30 22:29:02.822537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:02.822498    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:03.007004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.006964    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:03.170900 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:29:03.170863    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256a1079_f087_481a_98ad_745f95415d8f.slice/crio-5538332c39d151e6b27ed247308461fb494b54b2e2e25ad9dd7a7f180d9a57ef WatchSource:0}: Error finding container 5538332c39d151e6b27ed247308461fb494b54b2e2e25ad9dd7a7f180d9a57ef: Status 404 returned error can't find the container with id 5538332c39d151e6b27ed247308461fb494b54b2e2e25ad9dd7a7f180d9a57ef
Oct 30 22:29:03.171562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.171536    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"]
Oct 30 22:29:03.174115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.174083    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:03.174315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.174145    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:03.174315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.174176    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:03.276415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.276382    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:03.554183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.554141    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" event={"ID":"256a1079-f087-481a-98ad-745f95415d8f","Type":"ContainerStarted","Data":"9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"}
Oct 30 22:29:03.554868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.554802    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" event={"ID":"256a1079-f087-481a-98ad-745f95415d8f","Type":"ContainerStarted","Data":"5538332c39d151e6b27ed247308461fb494b54b2e2e25ad9dd7a7f180d9a57ef"}
Oct 30 22:29:03.678227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.678164    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podStartSLOduration=1.678150085 podStartE2EDuration="1.678150085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:03.67502875 +0000 UTC m=+1674.524541725" watchObservedRunningTime="2026-10-30 22:29:03.678150085 +0000 UTC m=+1674.527663058"
Oct 30 22:29:03.693967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:03.693937    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:04.699224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:04.699190    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:05.145495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:05.145449    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:05.577831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:05.577799    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:05.578943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:05.578919    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:29:05.842770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:05.842612    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:07.340923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.340886    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:07.566220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.557077    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:07.683038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.682915    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"]
Oct 30 22:29:07.683517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.683466    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" containerID="cri-o://918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf" gracePeriod=30
Oct 30 22:29:07.713235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.713167    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"]
Oct 30 22:29:07.713500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.713443    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" containerID="cri-o://bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460" gracePeriod=30
Oct 30 22:29:07.729171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.728929    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"]
Oct 30 22:29:07.738436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.738180    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" containerID="cri-o://57766c24ff4414f368653a887c96a85f306f8319a9b7c086ad360910e02b232a" gracePeriod=30
Oct 30 22:29:07.739326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.737287    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:07.830504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.822181    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"]
Oct 30 22:29:07.831840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.831805    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:07.907710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.903405    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-73735c-dockercfg-lsnjs\""
Oct 30 22:29:07.907710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.904393    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-73735c\""
Oct 30 22:29:07.932871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.918978    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:07.932871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.926139    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:07.932871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:07.926413    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxzd\" (UniqueName: \"kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd\") pod \"payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:08.100221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.077504    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxzd\" (UniqueName: \"kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd\") pod \"payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:08.100221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.077774    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:08.100221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.078912    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:08.100221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.098621    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"]
Oct 30 22:29:08.104665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.104189    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:08.104665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.104266    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:08.227330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.227235    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxzd\" (UniqueName: \"kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd\") pod \"payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:08.269462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.269377    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"]
Oct 30 22:29:08.269856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.269832    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" containerID="cri-o://a720fb40ecabf6a34839d6e2c884966aac4b4cf7647e2a90d204c448365bf601" gracePeriod=30
Oct 30 22:29:08.276640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.276580    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"]
Oct 30 22:29:08.277777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.277391    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" containerID="cri-o://715e19bbc6bb753d9a907e1b12f7d95cb013a416503bae99f1b42f95ca1c9970" gracePeriod=30
Oct 30 22:29:08.283021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.282543    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"]
Oct 30 22:29:08.283021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.282792    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy" containerID="cri-o://dd125ed3c92c98a3a3377ac4250ae290202c52658861437ffb729febcb0854bf" gracePeriod=30
Oct 30 22:29:08.297139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.297100    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"]
Oct 30 22:29:08.298083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.298050    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy" containerID="cri-o://4ee43571de6db7858bd3e1ae70f5d347f6fa0f12928ef25499df23ec0657a0c5" gracePeriod=30
Oct 30 22:29:08.437905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.437856    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:08.677437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.677392    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:08.677637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.677477    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:08.846641 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:29:08.846585    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ebb9a4_b0ac_438b_a5ee_c22b618658dd.slice/crio-6e0625f3b7528c985507cfc416c776e11e238e0f19966a9b28d7140a3b8b1626 WatchSource:0}: Error finding container 6e0625f3b7528c985507cfc416c776e11e238e0f19966a9b28d7140a3b8b1626: Status 404 returned error can't find the container with id 6e0625f3b7528c985507cfc416c776e11e238e0f19966a9b28d7140a3b8b1626
Oct 30 22:29:08.847966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:08.847785    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"]
Oct 30 22:29:09.052104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:09.052055    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:09.256592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:09.256529    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:09.620657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:09.620562    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz" event={"ID":"64ebb9a4-b0ac-438b-a5ee-c22b618658dd","Type":"ContainerStarted","Data":"2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1"}
Oct 30 22:29:09.620657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:09.620619    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz" event={"ID":"64ebb9a4-b0ac-438b-a5ee-c22b618658dd","Type":"ContainerStarted","Data":"6e0625f3b7528c985507cfc416c776e11e238e0f19966a9b28d7140a3b8b1626"}
Oct 30 22:29:09.621197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:09.620672    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:09.738019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:09.737414    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz" podStartSLOduration=2.737398023 podStartE2EDuration="2.737398023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:09.733113097 +0000 UTC m=+1680.582626071" watchObservedRunningTime="2026-10-30 22:29:09.737398023 +0000 UTC m=+1680.586910997"
Oct 30 22:29:10.397158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:10.397122    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:10.681529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:10.681450    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:10.681529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:10.681502    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:11.756535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:11.756498    5028 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:11.756951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:11.756557    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:12.908384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:12.908351    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:29:12.908925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:12.908411    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 22:29:13.020796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.020747    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:29:13.064039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.063991    5028 scope.go:122] "RemoveContainer" containerID="7524e44f8975ec5517db7f8ea26de596e269c37337d018ffa586e816f8144a31"
Oct 30 22:29:13.084759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.084733    5028 scope.go:122] "RemoveContainer" containerID="d67e20d20630101fdb9ef25b31bfe93d79b64dcacded3dc46ac1be4522f3db57"
Oct 30 22:29:13.096961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.096920    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097012    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097062    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097091    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097254    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097396 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097378    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097428    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097537    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.097595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.097568    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2\") pod \"a2ac3c25-1344-4a8b-9499-8305d610254c\" (UID: \"a2ac3c25-1344-4a8b-9499-8305d610254c\") "
Oct 30 22:29:13.099109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.099076    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.099256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.099128    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.099515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.099490    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:13.099812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.099794    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.100003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.099984    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.103386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.103357    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.103672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.103571    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:13.107210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.107169    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.107497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.107452    5028 scope.go:122] "RemoveContainer" containerID="a54fb226827d3396fe0b0a89b9fc0f13ee2560d7e393344ce354aca31dfd01d8"
Oct 30 22:29:13.108110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.108088    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2" pod "a2ac3c25-1344-4a8b-9499-8305d610254c" (UID: "a2ac3c25-1344-4a8b-9499-8305d610254c"). InnerVolumeSpecName "kube-api-access-p59x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.122938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.122760    5028 scope.go:122] "RemoveContainer" containerID="8265890a7ce07fee6d98b4c8316245ac1822a9970c656e76d3c9df9bb9a4ecfa"
Oct 30 22:29:13.141384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.141359    5028 scope.go:122] "RemoveContainer" containerID="363a9f3552bdbfe4137004e312b54d1c54ef03347402ac57ce7b4fba8adb1824"
Oct 30 22:29:13.141502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.141477    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:29:13.153024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.152995    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:29:13.153860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.153840    5028 scope.go:122] "RemoveContainer" containerID="57766c24ff4414f368653a887c96a85f306f8319a9b7c086ad360910e02b232a"
Oct 30 22:29:13.165559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.165535    5028 scope.go:122] "RemoveContainer" containerID="ed47b0e7095be4e5f54783d9026effe5b90ccdb0e4ac78a74a9f045838876957"
Oct 30 22:29:13.191890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.191194    5028 scope.go:122] "RemoveContainer" containerID="bf63dbc510d1050357b5ffef0ec0953e1df49da1bafd2b6f6508a2857dd71e7e"
Oct 30 22:29:13.202480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202452    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac3c25-1344-4a8b-9499-8305d610254c-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202481    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202495    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202510    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202523    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202535    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202548    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202563    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ac3c25-1344-4a8b-9499-8305d610254c-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.202670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.202577    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p59x2\" (UniqueName: \"kubernetes.io/projected/a2ac3c25-1344-4a8b-9499-8305d610254c-kube-api-access-p59x2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.209879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.209852    5028 scope.go:122] "RemoveContainer" containerID="147eda251f68ae597c51158c4edd59e9e4a5898ba478b12ad017e74b8fda67bf"
Oct 30 22:29:13.230304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.230278    5028 scope.go:122] "RemoveContainer" containerID="46f42031ea491b78e9f53d48983691a44b1af29fec04a5ab59a6cd5b8ea6283c"
Oct 30 22:29:13.242524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.242503    5028 scope.go:122] "RemoveContainer" containerID="d23544b7cbbbb8cf1e5e927bf581bfd81226e5a512bf174833395774db0b3d8a"
Oct 30 22:29:13.262424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.262402    5028 scope.go:122] "RemoveContainer" containerID="6e9d4ed333e2f3d725b41cc6dc8d95e7bb3edad9787d51f112e20e4974d4205b"
Oct 30 22:29:13.273866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.273841    5028 scope.go:122] "RemoveContainer" containerID="640d7825605ddacc7a1cf37080f42d3f27721a14d923af37259e4eb20d4d93c8"
Oct 30 22:29:13.286992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.286968    5028 scope.go:122] "RemoveContainer" containerID="73b811f043eaced974e137fe4b79d99158494893c6c2bb81f8ef7cf4bdf3d301"
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315271    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315328    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315380    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315511    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315601    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315632    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315676    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315761    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315793    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315896    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.315954    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316065    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316127    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316167    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316225    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316331    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket\") pod \"21a390ea-157d-4a39-be2f-a506f548304a\" (UID: \"21a390ea-157d-4a39-be2f-a506f548304a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316388    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.317012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.316490    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo\") pod \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\" (UID: \"800b1992-b5d7-4eda-a700-b6c3807ecf0a\") "
Oct 30 22:29:13.323325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.322671    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.323325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.323092    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:13.323929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.323621    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.325415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.325386    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.326619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.326596    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.328084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.328040    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.329717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.329537    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.330295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.329944    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:13.330295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.330278    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.330491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.330467    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.331435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.331265    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.331435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.331397    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.331608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.331446    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:13.334755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.334682    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.335081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.335055    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:13.335197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.335122    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.336533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.336438    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l" pod "800b1992-b5d7-4eda-a700-b6c3807ecf0a" (UID: "800b1992-b5d7-4eda-a700-b6c3807ecf0a"). InnerVolumeSpecName "kube-api-access-gtl9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.336897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.336861    5028 scope.go:122] "RemoveContainer" containerID="a63883d78825c346c23a794ebb16c4ffcf42aafdd1042500ec507404ec9e81b6"
Oct 30 22:29:13.338863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.338844    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl" pod "21a390ea-157d-4a39-be2f-a506f548304a" (UID: "21a390ea-157d-4a39-be2f-a506f548304a"). InnerVolumeSpecName "kube-api-access-9qvnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.428624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428593    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428623    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428633    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428643    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428652    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428661    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a390ea-157d-4a39-be2f-a506f548304a-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428673    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428698    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvnl\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-kube-api-access-9qvnl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428712    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a390ea-157d-4a39-be2f-a506f548304a-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428725    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtl9l\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-kube-api-access-gtl9l\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428739    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428752    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428763    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800b1992-b5d7-4eda-a700-b6c3807ecf0a-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428773    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428781    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a390ea-157d-4a39-be2f-a506f548304a-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428789    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428798    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a390ea-157d-4a39-be2f-a506f548304a-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.428876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.428806    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800b1992-b5d7-4eda-a700-b6c3807ecf0a-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.529897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.529867    5028 scope.go:122] "RemoveContainer" containerID="3290d1c9872132671c662d660fd04f18e10935a174194fac6611528ddf16a840"
Oct 30 22:29:13.590503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.590401    5028 scope.go:122] "RemoveContainer" containerID="c86abe0680fd452ac63e1be90a4152d5533102ff791acfbc512bc228179cb354"
Oct 30 22:29:13.631238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.629876    5028 scope.go:122] "RemoveContainer" containerID="520ca174e7c20f130192afd6808ab6f3ab13bc56fbb5e169bc00bf230e6ae79c"
Oct 30 22:29:13.660561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.660403    5028 scope.go:122] "RemoveContainer" containerID="39a89abd2a43a8509854f5474c92842462bfcc3736ec0589c5e6b63a568fcfce"
Oct 30 22:29:13.763896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.763562    5028 scope.go:122] "RemoveContainer" containerID="a0fee25067bfced559734b511bf0857b1f5acdc183cef3362e35bdfd2cc93bdb"
Oct 30 22:29:13.775811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.775782    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:29:13.782501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.782463    5028 generic.go:356] "Generic (PLEG): container finished" podID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerID="715e19bbc6bb753d9a907e1b12f7d95cb013a416503bae99f1b42f95ca1c9970" exitCode=0
Oct 30 22:29:13.782675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.782574    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" event={"ID":"5405bee9-16b0-463b-a2c9-ca83c4e629f8","Type":"ContainerDied","Data":"715e19bbc6bb753d9a907e1b12f7d95cb013a416503bae99f1b42f95ca1c9970"}
Oct 30 22:29:13.788218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.788176    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" event={"ID":"800b1992-b5d7-4eda-a700-b6c3807ecf0a","Type":"ContainerDied","Data":"57766c24ff4414f368653a887c96a85f306f8319a9b7c086ad360910e02b232a"}
Oct 30 22:29:13.788383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.788249    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs" event={"ID":"800b1992-b5d7-4eda-a700-b6c3807ecf0a","Type":"ContainerDied","Data":"40614e6ba0af55f851fd619c6c1cec286ea71abbfcadf4579a900dc547528e4b"}
Oct 30 22:29:13.788383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.788342    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"
Oct 30 22:29:13.800354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.800178    5028 generic.go:356] "Generic (PLEG): container finished" podID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerID="dd125ed3c92c98a3a3377ac4250ae290202c52658861437ffb729febcb0854bf" exitCode=0
Oct 30 22:29:13.800354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.800277    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" event={"ID":"be620ffb-2caf-45a0-8042-ec52b8c24227","Type":"ContainerDied","Data":"dd125ed3c92c98a3a3377ac4250ae290202c52658861437ffb729febcb0854bf"}
Oct 30 22:29:13.803262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.803232    5028 generic.go:356] "Generic (PLEG): container finished" podID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerID="4ee43571de6db7858bd3e1ae70f5d347f6fa0f12928ef25499df23ec0657a0c5" exitCode=0
Oct 30 22:29:13.803475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.803459    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5" event={"ID":"74927d31-8f63-428f-bfbd-a9c517a2f117","Type":"ContainerDied","Data":"4ee43571de6db7858bd3e1ae70f5d347f6fa0f12928ef25499df23ec0657a0c5"}
Oct 30 22:29:13.803640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.803627    5028 scope.go:122] "RemoveContainer" containerID="4ee43571de6db7858bd3e1ae70f5d347f6fa0f12928ef25499df23ec0657a0c5"
Oct 30 22:29:13.803909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.803894    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"
Oct 30 22:29:13.810051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.809910    5028 generic.go:356] "Generic (PLEG): container finished" podID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerID="a720fb40ecabf6a34839d6e2c884966aac4b4cf7647e2a90d204c448365bf601" exitCode=0
Oct 30 22:29:13.810051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.809989    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" event={"ID":"6fce23ee-ef26-43d8-90fb-be6d1252a2e2","Type":"ContainerDied","Data":"a720fb40ecabf6a34839d6e2c884966aac4b4cf7647e2a90d204c448365bf601"}
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.850874    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.850947    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851001    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851145    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851188    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851297    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851346    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851439    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.851508    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token\") pod \"74927d31-8f63-428f-bfbd-a9c517a2f117\" (UID: \"74927d31-8f63-428f-bfbd-a9c517a2f117\") "
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.853388    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.853465    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.853833    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.853677    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.854711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.854058    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.858496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.858414    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "kube-api-access-frhm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.859142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.859030    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.859142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.859057    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.859142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.859073    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.859142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.859087    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhm6\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-kube-api-access-frhm6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.859142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.859101    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.859142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.859114    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74927d31-8f63-428f-bfbd-a9c517a2f117-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.873416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.872271    5028 generic.go:356] "Generic (PLEG): container finished" podID="21a390ea-157d-4a39-be2f-a506f548304a" containerID="918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf" exitCode=0
Oct 30 22:29:13.873416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.872373    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" event={"ID":"21a390ea-157d-4a39-be2f-a506f548304a","Type":"ContainerDied","Data":"918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf"}
Oct 30 22:29:13.873416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.872421    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49" event={"ID":"21a390ea-157d-4a39-be2f-a506f548304a","Type":"ContainerDied","Data":"5429d00f64b654e7239816cef8d53a5545866f2b107161de21e5b632df769f94"}
Oct 30 22:29:13.873416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.872449    5028 scope.go:122] "RemoveContainer" containerID="918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf"
Oct 30 22:29:13.873416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.872582    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"
Oct 30 22:29:13.880917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.880067    5028 generic.go:356] "Generic (PLEG): container finished" podID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerID="bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460" exitCode=0
Oct 30 22:29:13.880917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.880131    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" event={"ID":"a2ac3c25-1344-4a8b-9499-8305d610254c","Type":"ContainerDied","Data":"bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460"}
Oct 30 22:29:13.880917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.880177    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6" event={"ID":"a2ac3c25-1344-4a8b-9499-8305d610254c","Type":"ContainerDied","Data":"b6f1a255b056e228315aa95ae8f6519ca52961eaca80e78a85ebe464a3b7f089"}
Oct 30 22:29:13.880917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.880278    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"
Oct 30 22:29:13.891034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.891000    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:13.891567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.891536    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:13.895252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.895208    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token" pod "74927d31-8f63-428f-bfbd-a9c517a2f117" (UID: "74927d31-8f63-428f-bfbd-a9c517a2f117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:13.963025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.962990    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.963025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.963026    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.963511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.963040    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74927d31-8f63-428f-bfbd-a9c517a2f117-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:13.966460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.966437    5028 scope.go:122] "RemoveContainer" containerID="918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf"
Oct 30 22:29:13.967151 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:29:13.967094    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf\": container with ID starting with 918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf not found: ID does not exist" containerID="918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf"
Oct 30 22:29:13.967151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.967133    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf"} err="failed to get container status \"918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf\": rpc error: code = NotFound desc = could not find container \"918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf\": container with ID starting with 918464ff7ca3bb8f76bc6508aa68bbfc31cc5c4c637dd6a12ddb25b1eb8749bf not found: ID does not exist"
Oct 30 22:29:13.967300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.967159    5028 scope.go:122] "RemoveContainer" containerID="bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460"
Oct 30 22:29:13.981679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.981649    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:29:13.985475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.985358    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:29:13.988266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.988219    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:29:13.995077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.995049    5028 scope.go:122] "RemoveContainer" containerID="bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460"
Oct 30 22:29:13.995498 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:29:13.995434    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460\": container with ID starting with bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460 not found: ID does not exist" containerID="bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460"
Oct 30 22:29:13.995498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:13.995474    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460"} err="failed to get container status \"bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460\": rpc error: code = NotFound desc = could not find container \"bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460\": container with ID starting with bdc7cf256251db0440799d8c47534876945b7c378f61d4030bc573b9746b9460 not found: ID does not exist"
Oct 30 22:29:14.060035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.059429    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"]
Oct 30 22:29:14.071865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.071834    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.071965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.071916    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.071965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.071959    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.072105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.071990    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.072105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072054    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.072171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072101    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.072206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072182    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072246    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072291    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.072422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072402    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072464    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072536    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072581    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.072660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072616    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072644    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072697    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.072794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072738    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.072794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072760    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.072794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072788    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo\") pod \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\" (UID: \"5405bee9-16b0-463b-a2c9-ca83c4e629f8\") "
Oct 30 22:29:14.073005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072821    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.073005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072850    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.073005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072891    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.073005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072913    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.073005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072940    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.073005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.072964    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.073318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.073014    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data\") pod \"be620ffb-2caf-45a0-8042-ec52b8c24227\" (UID: \"be620ffb-2caf-45a0-8042-ec52b8c24227\") "
Oct 30 22:29:14.073318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.073050    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket\") pod \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\" (UID: \"6fce23ee-ef26-43d8-90fb-be6d1252a2e2\") "
Oct 30 22:29:14.076656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.076623    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.076878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.076856    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.079958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.079745    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.080494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.080227    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.081032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.080974    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.081189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.081101    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.081310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.081220    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "kube-api-access-n64bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.081498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.081472    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.083855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.083136    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.083855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.083219    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.083855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.083539    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.089126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.088936    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-5f9zs"]
Oct 30 22:29:14.089126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.089035    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.089509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.089484    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.089973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.089951    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.090279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.090261    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.093371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.092927    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.093371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.093153    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.093371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.093338    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.094055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.093726    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.095809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.094223    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:14.099098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.098619    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.099098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.098669    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4" pod "be620ffb-2caf-45a0-8042-ec52b8c24227" (UID: "be620ffb-2caf-45a0-8042-ec52b8c24227"). InnerVolumeSpecName "kube-api-access-pmsq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.099098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.098859    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.100741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.099373    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo" pod "5405bee9-16b0-463b-a2c9-ca83c4e629f8" (UID: "5405bee9-16b0-463b-a2c9-ca83c4e629f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.105863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.105838    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "kube-api-access-f4zvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:14.110761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.110728    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:14.119978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.119934    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy" pod "6fce23ee-ef26-43d8-90fb-be6d1252a2e2" (UID: "6fce23ee-ef26-43d8-90fb-be6d1252a2e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:14.175432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175397    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be620ffb-2caf-45a0-8042-ec52b8c24227-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175437    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175453    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175468    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175484    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175499    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175513    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175528    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175542    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175557    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.175573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175572    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175586    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175599    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175612    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175626    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175641    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175655    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zvc\" (UniqueName: \"kubernetes.io/projected/6fce23ee-ef26-43d8-90fb-be6d1252a2e2-kube-api-access-f4zvc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175671    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175730    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175745    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175759    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175775    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5405bee9-16b0-463b-a2c9-ca83c4e629f8-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175789    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64bw\" (UniqueName: \"kubernetes.io/projected/5405bee9-16b0-463b-a2c9-ca83c4e629f8-kube-api-access-n64bw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175803    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175817    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be620ffb-2caf-45a0-8042-ec52b8c24227-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175832    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmsq4\" (UniqueName: \"kubernetes.io/projected/be620ffb-2caf-45a0-8042-ec52b8c24227-kube-api-access-pmsq4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.176291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.175846    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5405bee9-16b0-463b-a2c9-ca83c4e629f8-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:14.456391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.456235    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:14.516195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.516161    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"]
Oct 30 22:29:14.527791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.527753    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qjbc6"]
Oct 30 22:29:14.589285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.587828    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:29:14.598746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.597090    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:29:14.613711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.611929    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:14.655872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.655792    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"]
Oct 30 22:29:14.659246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.657032    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-hsk49"]
Oct 30 22:29:14.860138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.860099    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"]
Oct 30 22:29:14.860956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.860939    5028 cpu_manager.go:414] "containerMap: removing container" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy"
Oct 30 22:29:14.861022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.860958    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy"
Oct 30 22:29:14.861022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.860982    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy"
Oct 30 22:29:14.861022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.860989    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy"
Oct 30 22:29:14.861022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.860998    5028 cpu_manager.go:414] "containerMap: removing container" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy"
Oct 30 22:29:14.861022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861004    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861032    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861036    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861055    5028 cpu_manager.go:414] "containerMap: removing container" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861060    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861079    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861084    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861110    5028 cpu_manager.go:414] "containerMap: removing container" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy"
Oct 30 22:29:14.861185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861115    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861217    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861228    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861238    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861247    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861252    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861260    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" containerName="istio-proxy"
Oct 30 22:29:14.861412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.861267    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a390ea-157d-4a39-be2f-a506f548304a" containerName="istio-proxy"
Oct 30 22:29:14.867556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.867527    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:14.895466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.895430    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"]
Oct 30 22:29:14.906205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.900598    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-qm5r5"]
Oct 30 22:29:14.914740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.914658    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"
Oct 30 22:29:14.915052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.915017    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb" event={"ID":"5405bee9-16b0-463b-a2c9-ca83c4e629f8","Type":"ContainerDied","Data":"3f4105f6a2f914efd88dacec88bd23425fe739235fb9cfc4b861fc3e05663915"}
Oct 30 22:29:14.915202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.915171    5028 scope.go:122] "RemoveContainer" containerID="715e19bbc6bb753d9a907e1b12f7d95cb013a416503bae99f1b42f95ca1c9970"
Oct 30 22:29:14.932545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.924492    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw" event={"ID":"be620ffb-2caf-45a0-8042-ec52b8c24227","Type":"ContainerDied","Data":"e6d8af063f0abbca56ad55c6e913f36b60c470a6cf748396c3807359955696ab"}
Oct 30 22:29:14.932545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.924710    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"
Oct 30 22:29:14.932545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.927313    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a3600a\""
Oct 30 22:29:14.932545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.927507    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a3600a-dockercfg-hlvhk\""
Oct 30 22:29:14.946738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.946678    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6" event={"ID":"6fce23ee-ef26-43d8-90fb-be6d1252a2e2","Type":"ContainerDied","Data":"ad30514901f10c0e3855dfd05f2028726bae3af37ced485ea1e801e1c4179d49"}
Oct 30 22:29:14.946944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.946845    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"
Oct 30 22:29:14.950450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.950417    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"]
Oct 30 22:29:14.979871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.979845    5028 scope.go:122] "RemoveContainer" containerID="dd125ed3c92c98a3a3377ac4250ae290202c52658861437ffb729febcb0854bf"
Oct 30 22:29:14.995636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:14.995606    5028 scope.go:122] "RemoveContainer" containerID="a720fb40ecabf6a34839d6e2c884966aac4b4cf7647e2a90d204c448365bf601"
Oct 30 22:29:15.010407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.010002    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gh9\" (UniqueName: \"kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9\") pod \"payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.010407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.010179    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.077709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.077561    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"]
Oct 30 22:29:15.096732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.093772    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.116030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.115989    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"]
Oct 30 22:29:15.140821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.135968    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.140821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.136411    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gh9\" (UniqueName: \"kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9\") pod \"payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.140821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.137740    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.269553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.267068    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5pv8\" (UniqueName: \"kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8\") pod \"payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.269553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.267213    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.334067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.334036    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gh9\" (UniqueName: \"kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9\") pod \"payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.340456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.340417    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a390ea-157d-4a39-be2f-a506f548304a" path="/var/lib/kubelet/pods/21a390ea-157d-4a39-be2f-a506f548304a/volumes"
Oct 30 22:29:15.341255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.341215    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74927d31-8f63-428f-bfbd-a9c517a2f117" path="/var/lib/kubelet/pods/74927d31-8f63-428f-bfbd-a9c517a2f117/volumes"
Oct 30 22:29:15.341764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.341746    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800b1992-b5d7-4eda-a700-b6c3807ecf0a" path="/var/lib/kubelet/pods/800b1992-b5d7-4eda-a700-b6c3807ecf0a/volumes"
Oct 30 22:29:15.342650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.342635    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ac3c25-1344-4a8b-9499-8305d610254c" path="/var/lib/kubelet/pods/a2ac3c25-1344-4a8b-9499-8305d610254c/volumes"
Oct 30 22:29:15.370057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.370021    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pv8\" (UniqueName: \"kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8\") pod \"payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.370225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.370135    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.371234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.371203    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.397321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.397288    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"]
Oct 30 22:29:15.403319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.403292    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8h5jw"]
Oct 30 22:29:15.429407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.429368    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pv8\" (UniqueName: \"kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8\") pod \"payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") " pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.487651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.487612    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:15.508563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.508532    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"]
Oct 30 22:29:15.508774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.508749    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:15.515269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.515238    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-267z6"]
Oct 30 22:29:15.677845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.677813    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.763712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.759809    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"]
Oct 30 22:29:15.769040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.769001    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-8qfxb"]
Oct 30 22:29:15.881136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.881110    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"]
Oct 30 22:29:15.926121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.926090    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"]
Oct 30 22:29:15.933343 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:29:15.933310    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a6a05e_1d05_4a5d_a336_952dd731df02.slice/crio-cc5916220c002ba50ce18bc09723df2bf88b35a27b7016ee5bfab80a0a76efa9 WatchSource:0}: Error finding container cc5916220c002ba50ce18bc09723df2bf88b35a27b7016ee5bfab80a0a76efa9: Status 404 returned error can't find the container with id cc5916220c002ba50ce18bc09723df2bf88b35a27b7016ee5bfab80a0a76efa9
Oct 30 22:29:15.966992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.966947    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh" event={"ID":"41a6a05e-1d05-4a5d-a336-952dd731df02","Type":"ContainerStarted","Data":"cc5916220c002ba50ce18bc09723df2bf88b35a27b7016ee5bfab80a0a76efa9"}
Oct 30 22:29:15.968556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:15.968531    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr" event={"ID":"5b248468-49b4-4f41-a435-667efec10460","Type":"ContainerStarted","Data":"c98c22d3833098eb262290db13677ff6d57998aefe8241e7611801c0053a5daf"}
Oct 30 22:29:16.981000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:16.980963    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh" event={"ID":"41a6a05e-1d05-4a5d-a336-952dd731df02","Type":"ContainerStarted","Data":"75f1666e5f84d3bc9ba339b75796c10b7ef1051f38f394614453715cab327aa1"}
Oct 30 22:29:16.981515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:16.981192    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:16.982748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:16.982676    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr" event={"ID":"5b248468-49b4-4f41-a435-667efec10460","Type":"ContainerStarted","Data":"3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0"}
Oct 30 22:29:16.982855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:16.982777    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:17.094701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:17.094645    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh" podStartSLOduration=3.094630483 podStartE2EDuration="3.094630483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:17.092506467 +0000 UTC m=+1687.942019441" watchObservedRunningTime="2026-10-30 22:29:17.094630483 +0000 UTC m=+1687.944143503"
Oct 30 22:29:17.216843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:17.205906    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr" podStartSLOduration=3.205886381 podStartE2EDuration="3.205886381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:17.204024247 +0000 UTC m=+1688.053537220" watchObservedRunningTime="2026-10-30 22:29:17.205886381 +0000 UTC m=+1688.055399354"
Oct 30 22:29:17.296351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:17.296274    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5405bee9-16b0-463b-a2c9-ca83c4e629f8" path="/var/lib/kubelet/pods/5405bee9-16b0-463b-a2c9-ca83c4e629f8/volumes"
Oct 30 22:29:17.296894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:17.296879    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fce23ee-ef26-43d8-90fb-be6d1252a2e2" path="/var/lib/kubelet/pods/6fce23ee-ef26-43d8-90fb-be6d1252a2e2/volumes"
Oct 30 22:29:17.297364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:17.297351    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be620ffb-2caf-45a0-8042-ec52b8c24227" path="/var/lib/kubelet/pods/be620ffb-2caf-45a0-8042-ec52b8c24227/volumes"
Oct 30 22:29:20.656051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:20.656011    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:29:20.677737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:20.677700    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:20.867710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:20.865342    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:21.180890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:21.180851    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:24.510494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:24.510439    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.446184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:25.446114    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.635316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:25.635263    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:26.073569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:26.073513    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.996217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:27.996186    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:29:27.996741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:27.996285    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:29:28.438961 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:29:28.438858    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:29:30.146838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.146803    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:30.150467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.150432    5028 generic.go:356] "Generic (PLEG): container finished" podID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerID="e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68" exitCode=0
Oct 30 22:29:30.151423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.150504    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" event={"ID":"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7","Type":"ContainerDied","Data":"e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"}
Oct 30 22:29:30.151423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.151003    5028 scope.go:122] "RemoveContainer" containerID="e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"
Oct 30 22:29:30.153029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.153000    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:30.153147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.153069    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:30.153147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:30.153103    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:31.166279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:31.166233    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" event={"ID":"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7","Type":"ContainerStarted","Data":"94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601"}
Oct 30 22:29:31.366488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:31.366456    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:29:31.367744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:31.367660    5028 patch_prober.go:29] interesting pod/e2e-mt-61346ee8-openshift-default-c79649cc-r8659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:29:31.367899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:31.367758    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 22:29:32.366273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:32.366232    5028 patch_prober.go:29] interesting pod/e2e-mt-61346ee8-openshift-default-c79649cc-r8659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:29:32.366811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:32.366297    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 22:29:32.409503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:32.409460    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:33.366418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:33.366376    5028 patch_prober.go:29] interesting pod/e2e-mt-61346ee8-openshift-default-c79649cc-r8659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:29:33.366877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:33.366445    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 22:29:34.315096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:34.315060    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:29:34.366480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:34.366433    5028 patch_prober.go:29] interesting pod/e2e-mt-61346ee8-openshift-default-c79649cc-r8659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:29:34.366912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:34.366504    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 22:29:34.646717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:34.646604    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:35.366193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:35.366153    5028 patch_prober.go:29] interesting pod/e2e-mt-61346ee8-openshift-default-c79649cc-r8659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:29:35.366387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:35.366221    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 22:29:36.366465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:36.366431    5028 patch_prober.go:29] interesting pod/e2e-mt-61346ee8-openshift-default-c79649cc-r8659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:29:36.366967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:36.366492    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 30 22:29:37.370984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:37.370949    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:29:37.372227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:37.372204    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:29:39.150057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:39.150019    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:43.777558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:43.777513    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.148143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:48.148091    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.307839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:48.307804    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.758993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:48.758957    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:52.449113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.449082    5028 generic.go:356] "Generic (PLEG): container finished" podID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerID="4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9" exitCode=0
Oct 30 22:29:52.449618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.449155    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" event={"ID":"b1b27184-4cbd-46d1-ade8-c061d5da46a4","Type":"ContainerDied","Data":"4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"}
Oct 30 22:29:52.449725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.449671    5028 scope.go:122] "RemoveContainer" containerID="4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"
Oct 30 22:29:52.451929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.451890    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:52.452066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.451986    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:52.452066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.452032    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:52.871719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:52.869523    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.464040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:53.464003    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" event={"ID":"b1b27184-4cbd-46d1-ade8-c061d5da46a4","Type":"ContainerStarted","Data":"b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da"}
Oct 30 22:29:54.278465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:54.278433    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:29:54.279432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:54.279405    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:29:54.279577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:54.279456    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:29:55.278264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:55.278228    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:29:55.278758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:55.278590    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:29:56.278718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:56.278651    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:29:56.279161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:56.278758    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:29:57.277960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:57.277919    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:29:57.278156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:57.277977    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:29:57.571981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:57.571882    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:58.278293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:58.278252    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 22:29:58.278536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:58.278303    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 22:29:58.297828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:58.297796    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:29:58.467145 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:29:58.467093    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:29:59.288345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:59.288311    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:29:59.299041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:29:59.299015    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:30:00.173023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.172979    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"]
Oct 30 22:30:00.177171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.177141    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.178399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.178368    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"]
Oct 30 22:30:00.182386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.182365    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.182962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.182937    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"]
Oct 30 22:30:00.189730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.189680    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"]
Oct 30 22:30:00.189861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.189823    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.194201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.194181    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.194388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.194364    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"]
Oct 30 22:30:00.197972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.197952    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.224617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.224586    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"]
Oct 30 22:30:00.232090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.232063    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.232800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.232773    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"]
Oct 30 22:30:00.234136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.234101    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:30:00.234377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.234160    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:00.234377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.234182    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:00.246400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.246255    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"]
Oct 30 22:30:00.246732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.246668    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.251283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.251262    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"]
Oct 30 22:30:00.251415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.251367    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.256666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.256639    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"]
Oct 30 22:30:00.262335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.262309    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"]
Oct 30 22:30:00.268046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.268024    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.289889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.288899    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.289889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.289783    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.293706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.290114    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghpdz\" (UniqueName: \"kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.294076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.289190    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"]
Oct 30 22:30:00.299188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.299134    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"]
Oct 30 22:30:00.299458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.294019    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4tqx\" (UniqueName: \"kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.299621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.299605    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxsb\" (UniqueName: \"kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.299835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.299816    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tpv\" (UniqueName: \"kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.300016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.299999    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.300204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.300185    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.300352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.300336    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl9c\" (UniqueName: \"kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.301627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.301607    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-62wpx\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.301856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.301836    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.302025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302004    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmdgc\" (UniqueName: \"kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.302119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302063    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.302185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302132    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7s4g\" (UniqueName: \"kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g\") pod \"maas-api-key-cleanup-29889990-62wpx\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.302246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302165    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrsd\" (UniqueName: \"kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.302332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302313    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmsd\" (UniqueName: \"kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.302495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302477    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.302664 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.302634    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.308425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.307818    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"]
Oct 30 22:30:00.323816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.323786    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"]
Oct 30 22:30:00.335078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.335011    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"]
Oct 30 22:30:00.361975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.361916    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"]
Oct 30 22:30:00.369168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.369134    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"]
Oct 30 22:30:00.404380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404342    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl9c\" (UniqueName: \"kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.404565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404510    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-62wpx\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.404565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404549    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.404740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404587    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmdgc\" (UniqueName: \"kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.404740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404682    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.404863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404763    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7s4g\" (UniqueName: \"kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g\") pod \"maas-api-key-cleanup-29889990-62wpx\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.404863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404794    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrsd\" (UniqueName: \"kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.405009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404879    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmsd\" (UniqueName: \"kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.405009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.404999    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.405125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405040    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.405190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405149    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.405246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405232    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.405310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405272    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghpdz\" (UniqueName: \"kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.405369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405308    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4tqx\" (UniqueName: \"kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.405369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405337    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxsb\" (UniqueName: \"kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.405369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405365    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tpv\" (UniqueName: \"kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.405548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405410    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.405548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405457    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.405548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405471    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-62wpx\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.405734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405575    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.405984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405954    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.407103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.405472    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.407103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.406287    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.407103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.406288    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.407103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.406457    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.407103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.407042    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.407103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.407071    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.464186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.464144    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl9c\" (UniqueName: \"kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.464392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.464196    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmdgc\" (UniqueName: \"kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc\") pod \"maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.465122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.465092    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmsd\" (UniqueName: \"kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd\") pod \"maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.466439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.466399    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7s4g\" (UniqueName: \"kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g\") pod \"maas-api-key-cleanup-29889990-62wpx\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.466548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.466508    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tpv\" (UniqueName: \"kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.466718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.466669    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxsb\" (UniqueName: \"kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb\") pod \"maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.467071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.467052    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrsd\" (UniqueName: \"kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.467285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.467266    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4tqx\" (UniqueName: \"kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.467871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.467849    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghpdz\" (UniqueName: \"kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:00.494897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.494869    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:00.503651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.503626    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:00.543433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.543402    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:00.547656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.547623    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:00.551025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.550990    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:00.566983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.566768    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:00.586190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.585939    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:00.614708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.614499    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:00.618881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:00.618627    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:01.107615 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.107574    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf83d7a_15cb_4287_85a6_0f6b4e28dc4d.slice/crio-3e5945c2dc2bbda27c4a6852744f3d76317d38b96cbafeeb88ce65a2e8ad92a3 WatchSource:0}: Error finding container 3e5945c2dc2bbda27c4a6852744f3d76317d38b96cbafeeb88ce65a2e8ad92a3: Status 404 returned error can't find the container with id 3e5945c2dc2bbda27c4a6852744f3d76317d38b96cbafeeb88ce65a2e8ad92a3
Oct 30 22:30:01.108171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.108073    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"]
Oct 30 22:30:01.144928 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.144885    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4970e768_2625_4c3c_ac71_9eeeb2fbe3b5.slice/crio-e6f63bc3588575c5fa2a9de559eb0b1485783104aa2349728708ee29a63e3d5f WatchSource:0}: Error finding container e6f63bc3588575c5fa2a9de559eb0b1485783104aa2349728708ee29a63e3d5f: Status 404 returned error can't find the container with id e6f63bc3588575c5fa2a9de559eb0b1485783104aa2349728708ee29a63e3d5f
Oct 30 22:30:01.146217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.146185    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"]
Oct 30 22:30:01.154300 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.154261    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a85e83_9a2b_4994_a152_074b0815e69c.slice/crio-f87606c43dea95ec2bc81c59d6b996cffac8327fd99f7d1408422ca9c7cd17f9 WatchSource:0}: Error finding container f87606c43dea95ec2bc81c59d6b996cffac8327fd99f7d1408422ca9c7cd17f9: Status 404 returned error can't find the container with id f87606c43dea95ec2bc81c59d6b996cffac8327fd99f7d1408422ca9c7cd17f9
Oct 30 22:30:01.156387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.156355    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"]
Oct 30 22:30:01.161400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.161376    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"]
Oct 30 22:30:01.163273 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.163249    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5e07d1_d36a_447d_906b_6eec1de1cd44.slice/crio-23700e57b7077cddc65f911a3d235b1cf7e029bc88ec7cb8194af0284c8a8347 WatchSource:0}: Error finding container 23700e57b7077cddc65f911a3d235b1cf7e029bc88ec7cb8194af0284c8a8347: Status 404 returned error can't find the container with id 23700e57b7077cddc65f911a3d235b1cf7e029bc88ec7cb8194af0284c8a8347
Oct 30 22:30:01.204776 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.204668    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c61bab_8baf_4190_ba3c_f291b183dc69.slice/crio-1dfeb2de25ac019c53d2d8bffe6ba76ed35e4be38181a1d4afa060b06c46e0e7 WatchSource:0}: Error finding container 1dfeb2de25ac019c53d2d8bffe6ba76ed35e4be38181a1d4afa060b06c46e0e7: Status 404 returned error can't find the container with id 1dfeb2de25ac019c53d2d8bffe6ba76ed35e4be38181a1d4afa060b06c46e0e7
Oct 30 22:30:01.208431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.208407    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"]
Oct 30 22:30:01.241662 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.241629    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"]
Oct 30 22:30:01.277310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.277277    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"]
Oct 30 22:30:01.308864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.308835    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"]
Oct 30 22:30:01.309196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.308944    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"]
Oct 30 22:30:01.330846 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.330816    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d151563_f8aa_4459_ba60_f048d5978c9a.slice/crio-bde9b2ca51f568531725388053ba0865049037884a091d1479740c44d277dd78 WatchSource:0}: Error finding container bde9b2ca51f568531725388053ba0865049037884a091d1479740c44d277dd78: Status 404 returned error can't find the container with id bde9b2ca51f568531725388053ba0865049037884a091d1479740c44d277dd78
Oct 30 22:30:01.332645 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:01.332602    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17f7b66_0a4b_4b50_827f_51b66871f50b.slice/crio-61b7f62c83d60a566ae9b32d4f6c6dabbb3e2f88d97f70cc5dc55fdf6ea13781 WatchSource:0}: Error finding container 61b7f62c83d60a566ae9b32d4f6c6dabbb3e2f88d97f70cc5dc55fdf6ea13781: Status 404 returned error can't find the container with id 61b7f62c83d60a566ae9b32d4f6c6dabbb3e2f88d97f70cc5dc55fdf6ea13781
Oct 30 22:30:01.552638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.552594    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" event={"ID":"37a85e83-9a2b-4994-a152-074b0815e69c","Type":"ContainerStarted","Data":"f87606c43dea95ec2bc81c59d6b996cffac8327fd99f7d1408422ca9c7cd17f9"}
Oct 30 22:30:01.553952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.553916    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" event={"ID":"c7c61bab-8baf-4190-ba3c-f291b183dc69","Type":"ContainerStarted","Data":"1dfeb2de25ac019c53d2d8bffe6ba76ed35e4be38181a1d4afa060b06c46e0e7"}
Oct 30 22:30:01.555274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.555241    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" event={"ID":"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d","Type":"ContainerStarted","Data":"3e5945c2dc2bbda27c4a6852744f3d76317d38b96cbafeeb88ce65a2e8ad92a3"}
Oct 30 22:30:01.556557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.556528    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" event={"ID":"d17f7b66-0a4b-4b50-827f-51b66871f50b","Type":"ContainerStarted","Data":"61b7f62c83d60a566ae9b32d4f6c6dabbb3e2f88d97f70cc5dc55fdf6ea13781"}
Oct 30 22:30:01.557814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.557786    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerStarted","Data":"bde9b2ca51f568531725388053ba0865049037884a091d1479740c44d277dd78"}
Oct 30 22:30:01.559137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.559113    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" event={"ID":"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5","Type":"ContainerStarted","Data":"e6f63bc3588575c5fa2a9de559eb0b1485783104aa2349728708ee29a63e3d5f"}
Oct 30 22:30:01.560371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.560343    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerStarted","Data":"23700e57b7077cddc65f911a3d235b1cf7e029bc88ec7cb8194af0284c8a8347"}
Oct 30 22:30:01.561526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.561502    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerStarted","Data":"8dc1e2a62f64d7c5d6c8229e3f4f7d8efcbc632686f765c0b3203da994958567"}
Oct 30 22:30:01.562855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.562834    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" event={"ID":"bf357047-0d19-46fd-a69e-0c62b92039c6","Type":"ContainerStarted","Data":"82a50a0f102890e250419c1b5cdce611c4b2791c9b6480a70487e4bed7065653"}
Oct 30 22:30:01.991918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:01.991887    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.157749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.157671    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.314143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.314108    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.573835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.573740    5028 generic.go:356] "Generic (PLEG): container finished" podID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerID="2c1eea695cc7c0bf52ca88576d7c850160abf367ccc3b8193197afaf4cc82976" exitCode=60
Oct 30 22:30:02.573998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.573828    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" event={"ID":"d17f7b66-0a4b-4b50-827f-51b66871f50b","Type":"ContainerDied","Data":"2c1eea695cc7c0bf52ca88576d7c850160abf367ccc3b8193197afaf4cc82976"}
Oct 30 22:30:02.574181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.574162    5028 scope.go:122] "RemoveContainer" containerID="2c1eea695cc7c0bf52ca88576d7c850160abf367ccc3b8193197afaf4cc82976"
Oct 30 22:30:02.575727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.575703    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerStarted","Data":"c4558c3f3e3887bdb2ccda4ba6ebf2a7fad3cc087ee1d9ffa6c734f523506c99"}
Oct 30 22:30:02.577568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.577545    5028 generic.go:356] "Generic (PLEG): container finished" podID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerID="f2816494b358478db9af4fe6207794adde92b3a4391b36e91f5b7d2e11568ddd" exitCode=60
Oct 30 22:30:02.577738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.577642    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" event={"ID":"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5","Type":"ContainerDied","Data":"f2816494b358478db9af4fe6207794adde92b3a4391b36e91f5b7d2e11568ddd"}
Oct 30 22:30:02.577903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.577888    5028 scope.go:122] "RemoveContainer" containerID="f2816494b358478db9af4fe6207794adde92b3a4391b36e91f5b7d2e11568ddd"
Oct 30 22:30:02.579517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.579496    5028 generic.go:356] "Generic (PLEG): container finished" podID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerID="69dc9784cbbc8fa5a26604be73740e3ac634ef95d63438fbca1c93952f08b104" exitCode=60
Oct 30 22:30:02.579607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.579525    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerDied","Data":"69dc9784cbbc8fa5a26604be73740e3ac634ef95d63438fbca1c93952f08b104"}
Oct 30 22:30:02.579838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.579821    5028 scope.go:122] "RemoveContainer" containerID="69dc9784cbbc8fa5a26604be73740e3ac634ef95d63438fbca1c93952f08b104"
Oct 30 22:30:02.581280 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.581247    5028 generic.go:356] "Generic (PLEG): container finished" podID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerID="9b7975d0635b9a7b6e558172d217da1dec97d8d456328cd98ff03ac3b54ff6fc" exitCode=60
Oct 30 22:30:02.581396 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.581302    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerDied","Data":"9b7975d0635b9a7b6e558172d217da1dec97d8d456328cd98ff03ac3b54ff6fc"}
Oct 30 22:30:02.581527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.581512    5028 scope.go:122] "RemoveContainer" containerID="9b7975d0635b9a7b6e558172d217da1dec97d8d456328cd98ff03ac3b54ff6fc"
Oct 30 22:30:02.583066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.583044    5028 generic.go:356] "Generic (PLEG): container finished" podID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerID="472cbe1b291b4c5537e26d839457d63c4b3cc9d25625029c6b6ed63733207c0f" exitCode=60
Oct 30 22:30:02.583151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.583130    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" event={"ID":"bf357047-0d19-46fd-a69e-0c62b92039c6","Type":"ContainerDied","Data":"472cbe1b291b4c5537e26d839457d63c4b3cc9d25625029c6b6ed63733207c0f"}
Oct 30 22:30:02.583289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.583273    5028 scope.go:122] "RemoveContainer" containerID="472cbe1b291b4c5537e26d839457d63c4b3cc9d25625029c6b6ed63733207c0f"
Oct 30 22:30:02.584941 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.584919    5028 generic.go:356] "Generic (PLEG): container finished" podID="37a85e83-9a2b-4994-a152-074b0815e69c" containerID="0461a5e6e6901f7ce15bd8e627d25896bea52833445559ce06238e7948c35685" exitCode=60
Oct 30 22:30:02.585045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.584996    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" event={"ID":"37a85e83-9a2b-4994-a152-074b0815e69c","Type":"ContainerDied","Data":"0461a5e6e6901f7ce15bd8e627d25896bea52833445559ce06238e7948c35685"}
Oct 30 22:30:02.585171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.585158    5028 scope.go:122] "RemoveContainer" containerID="0461a5e6e6901f7ce15bd8e627d25896bea52833445559ce06238e7948c35685"
Oct 30 22:30:02.586884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.586865    5028 generic.go:356] "Generic (PLEG): container finished" podID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerID="ce94edcab2ae15f64005bf0c72f3231064ecce52b4a9424a08a74d6df41a5737" exitCode=60
Oct 30 22:30:02.586973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.586893    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" event={"ID":"c7c61bab-8baf-4190-ba3c-f291b183dc69","Type":"ContainerDied","Data":"ce94edcab2ae15f64005bf0c72f3231064ecce52b4a9424a08a74d6df41a5737"}
Oct 30 22:30:02.587110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.587096    5028 scope.go:122] "RemoveContainer" containerID="ce94edcab2ae15f64005bf0c72f3231064ecce52b4a9424a08a74d6df41a5737"
Oct 30 22:30:02.591044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.591019    5028 generic.go:356] "Generic (PLEG): container finished" podID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerID="64984483eaeb232d4fcc03d25f6433a68e5d6c134826d581578d40963d776c6b" exitCode=60
Oct 30 22:30:02.591139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.591066    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" event={"ID":"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d","Type":"ContainerDied","Data":"64984483eaeb232d4fcc03d25f6433a68e5d6c134826d581578d40963d776c6b"}
Oct 30 22:30:02.591340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.591315    5028 scope.go:122] "RemoveContainer" containerID="64984483eaeb232d4fcc03d25f6433a68e5d6c134826d581578d40963d776c6b"
Oct 30 22:30:02.797357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:02.797302    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" podStartSLOduration=2.797286823 podStartE2EDuration="2.797286823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.795983228 +0000 UTC m=+1733.645496200" watchObservedRunningTime="2026-10-30 22:30:02.797286823 +0000 UTC m=+1733.646799795"
Oct 30 22:30:03.603924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.603880    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerStarted","Data":"deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa"}
Oct 30 22:30:03.606346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.606310    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" event={"ID":"37a85e83-9a2b-4994-a152-074b0815e69c","Type":"ContainerStarted","Data":"44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723"}
Oct 30 22:30:03.608462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.608432    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" event={"ID":"c7c61bab-8baf-4190-ba3c-f291b183dc69","Type":"ContainerStarted","Data":"eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93"}
Oct 30 22:30:03.610586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.610563    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" event={"ID":"d17f7b66-0a4b-4b50-827f-51b66871f50b","Type":"ContainerStarted","Data":"6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88"}
Oct 30 22:30:03.612363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.612335    5028 generic.go:356] "Generic (PLEG): container finished" podID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerID="c4558c3f3e3887bdb2ccda4ba6ebf2a7fad3cc087ee1d9ffa6c734f523506c99" exitCode=60
Oct 30 22:30:03.612454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.612408    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerDied","Data":"c4558c3f3e3887bdb2ccda4ba6ebf2a7fad3cc087ee1d9ffa6c734f523506c99"}
Oct 30 22:30:03.612643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.612628    5028 scope.go:122] "RemoveContainer" containerID="c4558c3f3e3887bdb2ccda4ba6ebf2a7fad3cc087ee1d9ffa6c734f523506c99"
Oct 30 22:30:03.614601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.614581    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" event={"ID":"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5","Type":"ContainerStarted","Data":"0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455"}
Oct 30 22:30:03.616895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.616874    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerStarted","Data":"3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb"}
Oct 30 22:30:03.727226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.727165    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" podStartSLOduration=3.72714537 podStartE2EDuration="3.72714537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.722576946 +0000 UTC m=+1734.572089920" watchObservedRunningTime="2026-10-30 22:30:03.72714537 +0000 UTC m=+1734.576658344"
Oct 30 22:30:03.953843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:03.953785    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" podStartSLOduration=3.953769769 podStartE2EDuration="3.953769769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.952434454 +0000 UTC m=+1734.801947427" watchObservedRunningTime="2026-10-30 22:30:03.953769769 +0000 UTC m=+1734.803282741"
Oct 30 22:30:04.061042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.061018    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"]
Oct 30 22:30:04.066553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.066526    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264c0a20-lhs2s"]
Oct 30 22:30:04.071509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.071462    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" podStartSLOduration=4.071447553 podStartE2EDuration="4.071447553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.064256526 +0000 UTC m=+1734.913769501" watchObservedRunningTime="2026-10-30 22:30:04.071447553 +0000 UTC m=+1734.920960526"
Oct 30 22:30:04.181988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.181877    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" podStartSLOduration=4.181860251 podStartE2EDuration="4.181860251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.180261967 +0000 UTC m=+1735.029774939" watchObservedRunningTime="2026-10-30 22:30:04.181860251 +0000 UTC m=+1735.031373225"
Oct 30 22:30:04.299292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.299242    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" podStartSLOduration=4.299227191 podStartE2EDuration="4.299227191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.296860927 +0000 UTC m=+1735.146373901" watchObservedRunningTime="2026-10-30 22:30:04.299227191 +0000 UTC m=+1735.148740163"
Oct 30 22:30:04.412462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.412418    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" podStartSLOduration=4.412404166 podStartE2EDuration="4.412404166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.407166307 +0000 UTC m=+1735.256679282" watchObservedRunningTime="2026-10-30 22:30:04.412404166 +0000 UTC m=+1735.261917139"
Oct 30 22:30:04.629220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.629183    5028 generic.go:356] "Generic (PLEG): container finished" podID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerID="0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455" exitCode=60
Oct 30 22:30:04.629722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.629266    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" event={"ID":"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5","Type":"ContainerDied","Data":"0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455"}
Oct 30 22:30:04.629722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.629341    5028 scope.go:122] "RemoveContainer" containerID="f2816494b358478db9af4fe6207794adde92b3a4391b36e91f5b7d2e11568ddd"
Oct 30 22:30:04.629823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.629738    5028 scope.go:122] "RemoveContainer" containerID="0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455"
Oct 30 22:30:04.630191 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.630167    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5_odh-ai-gateway-infra(4970e768-2625-4c3c-ac71-9eeeb2fbe3b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5"
Oct 30 22:30:04.631672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.631610    5028 generic.go:356] "Generic (PLEG): container finished" podID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerID="3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb" exitCode=60
Oct 30 22:30:04.631800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.631681    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerDied","Data":"3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb"}
Oct 30 22:30:04.632382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.632137    5028 scope.go:122] "RemoveContainer" containerID="3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb"
Oct 30 22:30:04.632597 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.632571    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw_odh-ai-gateway-infra(1f5e07d1-d36a-447d-906b-6eec1de1cd44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44"
Oct 30 22:30:04.634015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.633994    5028 generic.go:356] "Generic (PLEG): container finished" podID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerID="deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa" exitCode=60
Oct 30 22:30:04.634120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.634083    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerDied","Data":"deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa"}
Oct 30 22:30:04.634401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.634383    5028 scope.go:122] "RemoveContainer" containerID="deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa"
Oct 30 22:30:04.634737 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.634718    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr_odh-ai-gateway-infra(6938ec33-e8f9-4e67-9ee1-284ff6f3fc72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72"
Oct 30 22:30:04.636305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.636279    5028 generic.go:356] "Generic (PLEG): container finished" podID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerID="8135da6ac8a04981c303b8cdb68f30786c998f1d9b304d799aecbf858cb055e0" exitCode=60
Oct 30 22:30:04.636402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.636381    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" event={"ID":"bf357047-0d19-46fd-a69e-0c62b92039c6","Type":"ContainerDied","Data":"8135da6ac8a04981c303b8cdb68f30786c998f1d9b304d799aecbf858cb055e0"}
Oct 30 22:30:04.636809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.636787    5028 scope.go:122] "RemoveContainer" containerID="8135da6ac8a04981c303b8cdb68f30786c998f1d9b304d799aecbf858cb055e0"
Oct 30 22:30:04.637266 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.637234    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4_odh-ai-gateway-infra(bf357047-0d19-46fd-a69e-0c62b92039c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6"
Oct 30 22:30:04.638944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.638922    5028 generic.go:356] "Generic (PLEG): container finished" podID="37a85e83-9a2b-4994-a152-074b0815e69c" containerID="44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723" exitCode=60
Oct 30 22:30:04.639044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.638968    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" event={"ID":"37a85e83-9a2b-4994-a152-074b0815e69c","Type":"ContainerDied","Data":"44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723"}
Oct 30 22:30:04.639297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.639276    5028 scope.go:122] "RemoveContainer" containerID="44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723"
Oct 30 22:30:04.639610 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.639590    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p_odh-ai-gateway-infra(37a85e83-9a2b-4994-a152-074b0815e69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" podUID="37a85e83-9a2b-4994-a152-074b0815e69c"
Oct 30 22:30:04.641508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.641484    5028 generic.go:356] "Generic (PLEG): container finished" podID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerID="eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93" exitCode=60
Oct 30 22:30:04.641597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.641552    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" event={"ID":"c7c61bab-8baf-4190-ba3c-f291b183dc69","Type":"ContainerDied","Data":"eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93"}
Oct 30 22:30:04.641898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.641877    5028 scope.go:122] "RemoveContainer" containerID="eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93"
Oct 30 22:30:04.642224 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.642189    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-62wpx_odh-ai-gateway-infra(c7c61bab-8baf-4190-ba3c-f291b183dc69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69"
Oct 30 22:30:04.643919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.643898    5028 generic.go:356] "Generic (PLEG): container finished" podID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerID="0baa85634733d33bd33ed590750fd76a363527f90c991f432b62952bb303c9d9" exitCode=60
Oct 30 22:30:04.644005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.643976    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" event={"ID":"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d","Type":"ContainerDied","Data":"0baa85634733d33bd33ed590750fd76a363527f90c991f432b62952bb303c9d9"}
Oct 30 22:30:04.644251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.644232    5028 scope.go:122] "RemoveContainer" containerID="0baa85634733d33bd33ed590750fd76a363527f90c991f432b62952bb303c9d9"
Oct 30 22:30:04.644529 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.644503    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc_odh-ai-gateway-infra(7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d"
Oct 30 22:30:04.646216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.646192    5028 generic.go:356] "Generic (PLEG): container finished" podID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerID="6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88" exitCode=60
Oct 30 22:30:04.646361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.646310    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" event={"ID":"d17f7b66-0a4b-4b50-827f-51b66871f50b","Type":"ContainerDied","Data":"6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88"}
Oct 30 22:30:04.646637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.646613    5028 scope.go:122] "RemoveContainer" containerID="6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88"
Oct 30 22:30:04.646909 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:04.646888    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262_odh-ai-gateway-infra(d17f7b66-0a4b-4b50-827f-51b66871f50b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b"
Oct 30 22:30:04.647801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.647566    5028 scope.go:122] "RemoveContainer" containerID="69dc9784cbbc8fa5a26604be73740e3ac634ef95d63438fbca1c93952f08b104"
Oct 30 22:30:04.648653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.648630    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerStarted","Data":"851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190"}
Oct 30 22:30:04.667872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.667851    5028 scope.go:122] "RemoveContainer" containerID="9b7975d0635b9a7b6e558172d217da1dec97d8d456328cd98ff03ac3b54ff6fc"
Oct 30 22:30:04.684599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.684573    5028 scope.go:122] "RemoveContainer" containerID="472cbe1b291b4c5537e26d839457d63c4b3cc9d25625029c6b6ed63733207c0f"
Oct 30 22:30:04.700291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.700262    5028 scope.go:122] "RemoveContainer" containerID="0461a5e6e6901f7ce15bd8e627d25896bea52833445559ce06238e7948c35685"
Oct 30 22:30:04.715466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.715442    5028 scope.go:122] "RemoveContainer" containerID="ce94edcab2ae15f64005bf0c72f3231064ecce52b4a9424a08a74d6df41a5737"
Oct 30 22:30:04.828083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.825822    5028 scope.go:122] "RemoveContainer" containerID="64984483eaeb232d4fcc03d25f6433a68e5d6c134826d581578d40963d776c6b"
Oct 30 22:30:04.899223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:04.899199    5028 scope.go:122] "RemoveContainer" containerID="2c1eea695cc7c0bf52ca88576d7c850160abf367ccc3b8193197afaf4cc82976"
Oct 30 22:30:05.294949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.294906    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02b6505-66d7-4b97-bf34-149e4853f1f7" path="/var/lib/kubelet/pods/d02b6505-66d7-4b97-bf34-149e4853f1f7/volumes"
Oct 30 22:30:05.661150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.661059    5028 scope.go:122] "RemoveContainer" containerID="44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723"
Oct 30 22:30:05.661570 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.661444    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p_odh-ai-gateway-infra(37a85e83-9a2b-4994-a152-074b0815e69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" podUID="37a85e83-9a2b-4994-a152-074b0815e69c"
Oct 30 22:30:05.663013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.662985    5028 scope.go:122] "RemoveContainer" containerID="eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93"
Oct 30 22:30:05.663360 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.663337    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-62wpx_odh-ai-gateway-infra(c7c61bab-8baf-4190-ba3c-f291b183dc69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69"
Oct 30 22:30:05.664794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.664770    5028 scope.go:122] "RemoveContainer" containerID="0baa85634733d33bd33ed590750fd76a363527f90c991f432b62952bb303c9d9"
Oct 30 22:30:05.665169 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.665146    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc_odh-ai-gateway-infra(7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d"
Oct 30 22:30:05.666490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.666465    5028 scope.go:122] "RemoveContainer" containerID="6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88"
Oct 30 22:30:05.666800 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.666780    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262_odh-ai-gateway-infra(d17f7b66-0a4b-4b50-827f-51b66871f50b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b"
Oct 30 22:30:05.668281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.668259    5028 generic.go:356] "Generic (PLEG): container finished" podID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerID="851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190" exitCode=60
Oct 30 22:30:05.668389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.668338    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerDied","Data":"851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190"}
Oct 30 22:30:05.668463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.668392    5028 scope.go:122] "RemoveContainer" containerID="c4558c3f3e3887bdb2ccda4ba6ebf2a7fad3cc087ee1d9ffa6c734f523506c99"
Oct 30 22:30:05.668523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.668503    5028 scope.go:122] "RemoveContainer" containerID="851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190"
Oct 30 22:30:05.668809 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.668779    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv_odh-ai-gateway-infra(5d151563-f8aa-4459-ba60-f048d5978c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a"
Oct 30 22:30:05.670510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.670490    5028 scope.go:122] "RemoveContainer" containerID="0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455"
Oct 30 22:30:05.670794 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.670771    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5_odh-ai-gateway-infra(4970e768-2625-4c3c-ac71-9eeeb2fbe3b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5"
Oct 30 22:30:05.672211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.672190    5028 scope.go:122] "RemoveContainer" containerID="3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb"
Oct 30 22:30:05.672470 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.672446    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw_odh-ai-gateway-infra(1f5e07d1-d36a-447d-906b-6eec1de1cd44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44"
Oct 30 22:30:05.673928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.673912    5028 scope.go:122] "RemoveContainer" containerID="deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa"
Oct 30 22:30:05.674140 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.674115    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr_odh-ai-gateway-infra(6938ec33-e8f9-4e67-9ee1-284ff6f3fc72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72"
Oct 30 22:30:05.675616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:05.675599    5028 scope.go:122] "RemoveContainer" containerID="8135da6ac8a04981c303b8cdb68f30786c998f1d9b304d799aecbf858cb055e0"
Oct 30 22:30:05.675859 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:05.675841    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4_odh-ai-gateway-infra(bf357047-0d19-46fd-a69e-0c62b92039c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6"
Oct 30 22:30:06.687346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:06.687316    5028 scope.go:122] "RemoveContainer" containerID="851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190"
Oct 30 22:30:06.687823 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:06.687675    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv_odh-ai-gateway-infra(5d151563-f8aa-4459-ba60-f048d5978c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a"
Oct 30 22:30:06.743334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:06.743303    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:09.391253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:09.391207    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.194373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:11.194338    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.361243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:11.361185    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.531842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:11.531803    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.189861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:14.189828    5028 scope.go:122] "RemoveContainer" containerID="b7f3763a4e9b79c20ad232711cd973f22eccb1c1668958c9d58d999ffc3b53b7"
Oct 30 22:30:14.201499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:14.201466    5028 scope.go:122] "RemoveContainer" containerID="2169a396bda8ce4f8e2b136eb3a5845094a3be04ef251c585ad8eeb1515e5c24"
Oct 30 22:30:14.211932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:14.211912    5028 scope.go:122] "RemoveContainer" containerID="452e3994f5a56d11f5c1a76c6a0c9e839af0193ac7b43f4f2d8eaa860d875cca"
Oct 30 22:30:14.225057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:14.225029    5028 scope.go:122] "RemoveContainer" containerID="b86b133b817ee784e521f8c872aa7ff355d29a6cef780453c42f984e87031ec0"
Oct 30 22:30:14.238281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:14.238254    5028 scope.go:122] "RemoveContainer" containerID="5a807bc000e5e53249fca87bc4c811ddedd98374ecf8e2b3346345b974c1025f"
Oct 30 22:30:14.284125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:14.284094    5028 scope.go:122] "RemoveContainer" containerID="8135da6ac8a04981c303b8cdb68f30786c998f1d9b304d799aecbf858cb055e0"
Oct 30 22:30:15.296329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.296301    5028 scope.go:122] "RemoveContainer" containerID="3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb"
Oct 30 22:30:15.800864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.800829    5028 generic.go:356] "Generic (PLEG): container finished" podID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerID="abcfd94fbaefc79e6193a36dc8a2b856b7a150c658a0bf3293c80963f2a7db72" exitCode=60
Oct 30 22:30:15.801038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.800902    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" event={"ID":"bf357047-0d19-46fd-a69e-0c62b92039c6","Type":"ContainerDied","Data":"abcfd94fbaefc79e6193a36dc8a2b856b7a150c658a0bf3293c80963f2a7db72"}
Oct 30 22:30:15.801038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.800963    5028 scope.go:122] "RemoveContainer" containerID="8135da6ac8a04981c303b8cdb68f30786c998f1d9b304d799aecbf858cb055e0"
Oct 30 22:30:15.801444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.801420    5028 scope.go:122] "RemoveContainer" containerID="abcfd94fbaefc79e6193a36dc8a2b856b7a150c658a0bf3293c80963f2a7db72"
Oct 30 22:30:15.801877 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:15.801850    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4_odh-ai-gateway-infra(bf357047-0d19-46fd-a69e-0c62b92039c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6"
Oct 30 22:30:15.803194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.803173    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerStarted","Data":"a9d8e7454ddbdff3d804d6c871e2f329343560d99c9b8ba8c8de7781474acd7b"}
Oct 30 22:30:15.958946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:15.958909    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:16.284163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.284127    5028 scope.go:122] "RemoveContainer" containerID="6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88"
Oct 30 22:30:16.284506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.284487    5028 scope.go:122] "RemoveContainer" containerID="44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723"
Oct 30 22:30:16.284660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.284552    5028 scope.go:122] "RemoveContainer" containerID="eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93"
Oct 30 22:30:16.814681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.814647    5028 generic.go:356] "Generic (PLEG): container finished" podID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerID="a9d8e7454ddbdff3d804d6c871e2f329343560d99c9b8ba8c8de7781474acd7b" exitCode=60
Oct 30 22:30:16.815166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.814737    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerDied","Data":"a9d8e7454ddbdff3d804d6c871e2f329343560d99c9b8ba8c8de7781474acd7b"}
Oct 30 22:30:16.815166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.814800    5028 scope.go:122] "RemoveContainer" containerID="3c81f04458f49afd58360412f9019bb64def49721bea10cff728037b981f83cb"
Oct 30 22:30:16.815295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.815253    5028 scope.go:122] "RemoveContainer" containerID="a9d8e7454ddbdff3d804d6c871e2f329343560d99c9b8ba8c8de7781474acd7b"
Oct 30 22:30:16.815718 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:16.815676    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw_odh-ai-gateway-infra(1f5e07d1-d36a-447d-906b-6eec1de1cd44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44"
Oct 30 22:30:16.929127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:16.929044    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"]
Oct 30 22:30:17.057324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.057289    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"]
Oct 30 22:30:17.228401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.228367    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:17.296907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.296883    5028 scope.go:122] "RemoveContainer" containerID="851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190"
Oct 30 22:30:17.297068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.296974    5028 scope.go:122] "RemoveContainer" containerID="0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455"
Oct 30 22:30:17.347081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.347045    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb\" (UniqueName: \"kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb\") pod \"bf357047-0d19-46fd-a69e-0c62b92039c6\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") "
Oct 30 22:30:17.347249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.347166    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca\") pod \"bf357047-0d19-46fd-a69e-0c62b92039c6\" (UID: \"bf357047-0d19-46fd-a69e-0c62b92039c6\") "
Oct 30 22:30:17.348059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.348032    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca" pod "bf357047-0d19-46fd-a69e-0c62b92039c6" (UID: "bf357047-0d19-46fd-a69e-0c62b92039c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.358996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.358963    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb" pod "bf357047-0d19-46fd-a69e-0c62b92039c6" (UID: "bf357047-0d19-46fd-a69e-0c62b92039c6"). InnerVolumeSpecName "kube-api-access-bfxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.450736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.450610    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxsb\" (UniqueName: \"kubernetes.io/projected/bf357047-0d19-46fd-a69e-0c62b92039c6-kube-api-access-bfxsb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.450736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.450650    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf357047-0d19-46fd-a69e-0c62b92039c6-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.829770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.829729    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"
Oct 30 22:30:17.830291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.829722    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4" event={"ID":"bf357047-0d19-46fd-a69e-0c62b92039c6","Type":"ContainerDied","Data":"82a50a0f102890e250419c1b5cdce611c4b2791c9b6480a70487e4bed7065653"}
Oct 30 22:30:17.830291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.829960    5028 scope.go:122] "RemoveContainer" containerID="abcfd94fbaefc79e6193a36dc8a2b856b7a150c658a0bf3293c80963f2a7db72"
Oct 30 22:30:17.832347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.832323    5028 generic.go:356] "Generic (PLEG): container finished" podID="37a85e83-9a2b-4994-a152-074b0815e69c" containerID="7d7e6ab349594bb6a689bc206b67ca431973a4feb0607b41d0d379e64dbe256b" exitCode=60
Oct 30 22:30:17.832471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.832356    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" event={"ID":"37a85e83-9a2b-4994-a152-074b0815e69c","Type":"ContainerDied","Data":"7d7e6ab349594bb6a689bc206b67ca431973a4feb0607b41d0d379e64dbe256b"}
Oct 30 22:30:17.832959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.832934    5028 scope.go:122] "RemoveContainer" containerID="7d7e6ab349594bb6a689bc206b67ca431973a4feb0607b41d0d379e64dbe256b"
Oct 30 22:30:17.833494 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:17.833465    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p_odh-ai-gateway-infra(37a85e83-9a2b-4994-a152-074b0815e69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" podUID="37a85e83-9a2b-4994-a152-074b0815e69c"
Oct 30 22:30:17.834901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.834876    5028 generic.go:356] "Generic (PLEG): container finished" podID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerID="3a261249036ca94521a8c7429352024296e54706131250dc0633fcf6789556a5" exitCode=60
Oct 30 22:30:17.834990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.834973    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" event={"ID":"c7c61bab-8baf-4190-ba3c-f291b183dc69","Type":"ContainerDied","Data":"3a261249036ca94521a8c7429352024296e54706131250dc0633fcf6789556a5"}
Oct 30 22:30:17.835434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.835416    5028 scope.go:122] "RemoveContainer" containerID="3a261249036ca94521a8c7429352024296e54706131250dc0633fcf6789556a5"
Oct 30 22:30:17.835856 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:17.835796    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-62wpx_odh-ai-gateway-infra(c7c61bab-8baf-4190-ba3c-f291b183dc69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69"
Oct 30 22:30:17.837487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.837468    5028 generic.go:356] "Generic (PLEG): container finished" podID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerID="9044bf0c5512ef5e9c6d1fdb0083f2b922b39079cf234f0af3ff38c394b1ddee" exitCode=60
Oct 30 22:30:17.837595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.837497    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" event={"ID":"d17f7b66-0a4b-4b50-827f-51b66871f50b","Type":"ContainerDied","Data":"9044bf0c5512ef5e9c6d1fdb0083f2b922b39079cf234f0af3ff38c394b1ddee"}
Oct 30 22:30:17.837991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.837972    5028 scope.go:122] "RemoveContainer" containerID="9044bf0c5512ef5e9c6d1fdb0083f2b922b39079cf234f0af3ff38c394b1ddee"
Oct 30 22:30:17.838373 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:17.838329    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262_odh-ai-gateway-infra(d17f7b66-0a4b-4b50-827f-51b66871f50b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b"
Oct 30 22:30:17.844212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.844186    5028 scope.go:122] "RemoveContainer" containerID="44d3f496c159de35a2435217df5dea4a89ff03c35e3974c825dc2e5356484723"
Oct 30 22:30:17.873798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.873775    5028 scope.go:122] "RemoveContainer" containerID="eb2418fe691209b25961fed0f0c402d9900e2633d1fe14a72f2d458ea5b7fb93"
Oct 30 22:30:17.889869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:17.889840    5028 scope.go:122] "RemoveContainer" containerID="6105e3354d5ffdb1e2897d295ebc67fad524ada1e26d10b91549509ffd98ee88"
Oct 30 22:30:18.004454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.004420    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"]
Oct 30 22:30:18.009933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.009902    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-be17e7-29889990-rrgq4"]
Oct 30 22:30:18.036227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.036191    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:18.161484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.161444    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca\") pod \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") "
Oct 30 22:30:18.161674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.161658    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz\" (UniqueName: \"kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz\") pod \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\" (UID: \"1f5e07d1-d36a-447d-906b-6eec1de1cd44\") "
Oct 30 22:30:18.162188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.162157    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca" pod "1f5e07d1-d36a-447d-906b-6eec1de1cd44" (UID: "1f5e07d1-d36a-447d-906b-6eec1de1cd44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.162601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.162577    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5e07d1-d36a-447d-906b-6eec1de1cd44-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.164801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.164772    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz" pod "1f5e07d1-d36a-447d-906b-6eec1de1cd44" (UID: "1f5e07d1-d36a-447d-906b-6eec1de1cd44"). InnerVolumeSpecName "kube-api-access-ghpdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.264417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.264323    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghpdz\" (UniqueName: \"kubernetes.io/projected/1f5e07d1-d36a-447d-906b-6eec1de1cd44-kube-api-access-ghpdz\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.284050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.284020    5028 scope.go:122] "RemoveContainer" containerID="deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa"
Oct 30 22:30:18.858711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.858603    5028 generic.go:356] "Generic (PLEG): container finished" podID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerID="e601d11c761ace78f633dcf7c040976d2e5b58fc90cb7fc00372beda498a018d" exitCode=60
Oct 30 22:30:18.858711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.858679    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerDied","Data":"e601d11c761ace78f633dcf7c040976d2e5b58fc90cb7fc00372beda498a018d"}
Oct 30 22:30:18.859217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.858779    5028 scope.go:122] "RemoveContainer" containerID="851bddc485a626ce0f540355b4c5901f3c51b8c56b1853e86b63a8f9843b3190"
Oct 30 22:30:18.859302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.859231    5028 scope.go:122] "RemoveContainer" containerID="e601d11c761ace78f633dcf7c040976d2e5b58fc90cb7fc00372beda498a018d"
Oct 30 22:30:18.859697 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:18.859655    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv_odh-ai-gateway-infra(5d151563-f8aa-4459-ba60-f048d5978c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a"
Oct 30 22:30:18.862146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.861169    5028 generic.go:356] "Generic (PLEG): container finished" podID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerID="23828e049de81846168955cfcd83a1741acccac9b02b86ff70ab16c8bdc2a161" exitCode=60
Oct 30 22:30:18.862146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.861266    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" event={"ID":"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5","Type":"ContainerDied","Data":"23828e049de81846168955cfcd83a1741acccac9b02b86ff70ab16c8bdc2a161"}
Oct 30 22:30:18.862146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.861745    5028 scope.go:122] "RemoveContainer" containerID="23828e049de81846168955cfcd83a1741acccac9b02b86ff70ab16c8bdc2a161"
Oct 30 22:30:18.862146 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:18.862106    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5_odh-ai-gateway-infra(4970e768-2625-4c3c-ac71-9eeeb2fbe3b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5"
Oct 30 22:30:18.864371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.864242    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"
Oct 30 22:30:18.864371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.864256    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw" event={"ID":"1f5e07d1-d36a-447d-906b-6eec1de1cd44","Type":"ContainerDied","Data":"23700e57b7077cddc65f911a3d235b1cf7e029bc88ec7cb8194af0284c8a8347"}
Oct 30 22:30:18.867053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.867022    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerStarted","Data":"d8a5eed66c599909c9c3e0919dd37c6f4ecb6d22ba5ebe30b8ec6ee105d78f16"}
Oct 30 22:30:18.879821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.879802    5028 scope.go:122] "RemoveContainer" containerID="0c99097df6781b4b7378f2dfce876acc9f2797f8c8b4a27997fcba6efdd98455"
Oct 30 22:30:18.893576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:18.893556    5028 scope.go:122] "RemoveContainer" containerID="a9d8e7454ddbdff3d804d6c871e2f329343560d99c9b8ba8c8de7781474acd7b"
Oct 30 22:30:19.085216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.085176    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"]
Oct 30 22:30:19.090119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.090089    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889990-5ptxw"]
Oct 30 22:30:19.128665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.128575    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"]
Oct 30 22:30:19.246071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.246029    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"]
Oct 30 22:30:19.286522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.286054    5028 scope.go:122] "RemoveContainer" containerID="0baa85634733d33bd33ed590750fd76a363527f90c991f432b62952bb303c9d9"
Oct 30 22:30:19.305240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.305205    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" path="/var/lib/kubelet/pods/1f5e07d1-d36a-447d-906b-6eec1de1cd44/volumes"
Oct 30 22:30:19.305871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.305830    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" path="/var/lib/kubelet/pods/bf357047-0d19-46fd-a69e-0c62b92039c6/volumes"
Oct 30 22:30:19.376475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.376443    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:19.459079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.459047    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"]
Oct 30 22:30:19.489103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.489071    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca\") pod \"d17f7b66-0a4b-4b50-827f-51b66871f50b\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") "
Oct 30 22:30:19.489289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.489269    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd\" (UniqueName: \"kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd\") pod \"d17f7b66-0a4b-4b50-827f-51b66871f50b\" (UID: \"d17f7b66-0a4b-4b50-827f-51b66871f50b\") "
Oct 30 22:30:19.489488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.489467    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca" pod "d17f7b66-0a4b-4b50-827f-51b66871f50b" (UID: "d17f7b66-0a4b-4b50-827f-51b66871f50b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.489858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.489839    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17f7b66-0a4b-4b50-827f-51b66871f50b-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.491429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.491405    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd" pod "d17f7b66-0a4b-4b50-827f-51b66871f50b" (UID: "d17f7b66-0a4b-4b50-827f-51b66871f50b"). InnerVolumeSpecName "kube-api-access-zlrsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.522788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.522659    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:19.591174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.591142    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrsd\" (UniqueName: \"kubernetes.io/projected/d17f7b66-0a4b-4b50-827f-51b66871f50b-kube-api-access-zlrsd\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.670946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.670919    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:19.692295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.692207    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca\") pod \"c7c61bab-8baf-4190-ba3c-f291b183dc69\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") "
Oct 30 22:30:19.692456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.692405    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g\" (UniqueName: \"kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g\") pod \"c7c61bab-8baf-4190-ba3c-f291b183dc69\" (UID: \"c7c61bab-8baf-4190-ba3c-f291b183dc69\") "
Oct 30 22:30:19.692638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.692611    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca" pod "c7c61bab-8baf-4190-ba3c-f291b183dc69" (UID: "c7c61bab-8baf-4190-ba3c-f291b183dc69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.693015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.692996    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c61bab-8baf-4190-ba3c-f291b183dc69-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.694753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.694724    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g" pod "c7c61bab-8baf-4190-ba3c-f291b183dc69" (UID: "c7c61bab-8baf-4190-ba3c-f291b183dc69"). InnerVolumeSpecName "kube-api-access-g7s4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.794412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.794374    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx\" (UniqueName: \"kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx\") pod \"37a85e83-9a2b-4994-a152-074b0815e69c\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") "
Oct 30 22:30:19.794590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.794514    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca\") pod \"37a85e83-9a2b-4994-a152-074b0815e69c\" (UID: \"37a85e83-9a2b-4994-a152-074b0815e69c\") "
Oct 30 22:30:19.794940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.794904    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca" pod "37a85e83-9a2b-4994-a152-074b0815e69c" (UID: "37a85e83-9a2b-4994-a152-074b0815e69c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.795317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.795298    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a85e83-9a2b-4994-a152-074b0815e69c-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.795317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.795319    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7s4g\" (UniqueName: \"kubernetes.io/projected/c7c61bab-8baf-4190-ba3c-f291b183dc69-kube-api-access-g7s4g\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.817236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.817194    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx" pod "37a85e83-9a2b-4994-a152-074b0815e69c" (UID: "37a85e83-9a2b-4994-a152-074b0815e69c"). InnerVolumeSpecName "kube-api-access-t4tqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.891372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.891334    5028 generic.go:356] "Generic (PLEG): container finished" podID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerID="d8a5eed66c599909c9c3e0919dd37c6f4ecb6d22ba5ebe30b8ec6ee105d78f16" exitCode=60
Oct 30 22:30:19.891878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.891673    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerDied","Data":"d8a5eed66c599909c9c3e0919dd37c6f4ecb6d22ba5ebe30b8ec6ee105d78f16"}
Oct 30 22:30:19.891878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.891799    5028 scope.go:122] "RemoveContainer" containerID="deb368c9ce5797b1f1a3fa2ba818eb26df5347f26fd659cc947a7669ba85a4fa"
Oct 30 22:30:19.892405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.892386    5028 scope.go:122] "RemoveContainer" containerID="d8a5eed66c599909c9c3e0919dd37c6f4ecb6d22ba5ebe30b8ec6ee105d78f16"
Oct 30 22:30:19.892979 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:19.892954    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr_odh-ai-gateway-infra(6938ec33-e8f9-4e67-9ee1-284ff6f3fc72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72"
Oct 30 22:30:19.897362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.897340    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4tqx\" (UniqueName: \"kubernetes.io/projected/37a85e83-9a2b-4994-a152-074b0815e69c-kube-api-access-t4tqx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.898211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.898185    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"
Oct 30 22:30:19.898300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.898275    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p" event={"ID":"37a85e83-9a2b-4994-a152-074b0815e69c","Type":"ContainerDied","Data":"f87606c43dea95ec2bc81c59d6b996cffac8327fd99f7d1408422ca9c7cd17f9"}
Oct 30 22:30:19.907718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.907663    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx" event={"ID":"c7c61bab-8baf-4190-ba3c-f291b183dc69","Type":"ContainerDied","Data":"1dfeb2de25ac019c53d2d8bffe6ba76ed35e4be38181a1d4afa060b06c46e0e7"}
Oct 30 22:30:19.907873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.907852    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"
Oct 30 22:30:19.912106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.912078    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" event={"ID":"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d","Type":"ContainerStarted","Data":"c0618b519925f7b0148fb0ba07c7c2d64a21b956635855fe28d38e7af35d47df"}
Oct 30 22:30:19.914234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.914210    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262" event={"ID":"d17f7b66-0a4b-4b50-827f-51b66871f50b","Type":"ContainerDied","Data":"61b7f62c83d60a566ae9b32d4f6c6dabbb3e2f88d97f70cc5dc55fdf6ea13781"}
Oct 30 22:30:19.914333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.914315    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"
Oct 30 22:30:19.916620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.916603    5028 scope.go:122] "RemoveContainer" containerID="7d7e6ab349594bb6a689bc206b67ca431973a4feb0607b41d0d379e64dbe256b"
Oct 30 22:30:19.944733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.944663    5028 scope.go:122] "RemoveContainer" containerID="3a261249036ca94521a8c7429352024296e54706131250dc0633fcf6789556a5"
Oct 30 22:30:19.959231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.959206    5028 scope.go:122] "RemoveContainer" containerID="9044bf0c5512ef5e9c6d1fdb0083f2b922b39079cf234f0af3ff38c394b1ddee"
Oct 30 22:30:19.987740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:19.987704    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"]
Oct 30 22:30:20.107776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.107738    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"]
Oct 30 22:30:20.114313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.114273    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bbd10f-29889990-l9p4p"]
Oct 30 22:30:20.175875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.175842    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:20.209090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.209066    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"]
Oct 30 22:30:20.306529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.304961    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca\") pod \"5d151563-f8aa-4459-ba60-f048d5978c9a\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") "
Oct 30 22:30:20.306529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.305174    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv\" (UniqueName: \"kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv\") pod \"5d151563-f8aa-4459-ba60-f048d5978c9a\" (UID: \"5d151563-f8aa-4459-ba60-f048d5978c9a\") "
Oct 30 22:30:20.306529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.306485    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca" pod "5d151563-f8aa-4459-ba60-f048d5978c9a" (UID: "5d151563-f8aa-4459-ba60-f048d5978c9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.310417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.310333    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv" pod "5d151563-f8aa-4459-ba60-f048d5978c9a" (UID: "5d151563-f8aa-4459-ba60-f048d5978c9a"). InnerVolumeSpecName "kube-api-access-l2tpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.328112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.327017    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"]
Oct 30 22:30:20.330812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.330264    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" podStartSLOduration=20.330222654 podStartE2EDuration="20.330222654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:20.329615411 +0000 UTC m=+1751.179128389" watchObservedRunningTime="2026-10-30 22:30:20.330222654 +0000 UTC m=+1751.179735629"
Oct 30 22:30:20.407657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.407620    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d151563-f8aa-4459-ba60-f048d5978c9a-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.407657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.407656    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tpv\" (UniqueName: \"kubernetes.io/projected/5d151563-f8aa-4459-ba60-f048d5978c9a-kube-api-access-l2tpv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.409752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.409648    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:20.469949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.469850    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"]
Oct 30 22:30:20.476195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.476031    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7c5d18-29889990-wc262"]
Oct 30 22:30:20.511018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.509837    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca\") pod \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") "
Oct 30 22:30:20.511018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.510040    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c\" (UniqueName: \"kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c\") pod \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\" (UID: \"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5\") "
Oct 30 22:30:20.511018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.510978    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca" pod "4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" (UID: "4970e768-2625-4c3c-ac71-9eeeb2fbe3b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.516095 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.516061    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c" pod "4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" (UID: "4970e768-2625-4c3c-ac71-9eeeb2fbe3b5"). InnerVolumeSpecName "kube-api-access-4pl9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.520159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.520089    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:20.597838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.597748    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"]
Oct 30 22:30:20.618420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.618381    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.618420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.618421    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl9c\" (UniqueName: \"kubernetes.io/projected/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5-kube-api-access-4pl9c\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.620300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.620212    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-62wpx"]
Oct 30 22:30:20.684611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.684574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:20.890448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.890313    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:20.950911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.950859    5028 generic.go:356] "Generic (PLEG): container finished" podID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerID="c0618b519925f7b0148fb0ba07c7c2d64a21b956635855fe28d38e7af35d47df" exitCode=60
Oct 30 22:30:20.951270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.950930    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" event={"ID":"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d","Type":"ContainerDied","Data":"c0618b519925f7b0148fb0ba07c7c2d64a21b956635855fe28d38e7af35d47df"}
Oct 30 22:30:20.951270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.951002    5028 scope.go:122] "RemoveContainer" containerID="0baa85634733d33bd33ed590750fd76a363527f90c991f432b62952bb303c9d9"
Oct 30 22:30:20.951459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.951439    5028 scope.go:122] "RemoveContainer" containerID="c0618b519925f7b0148fb0ba07c7c2d64a21b956635855fe28d38e7af35d47df"
Oct 30 22:30:20.951928 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:20.951888    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc_odh-ai-gateway-infra(7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d"
Oct 30 22:30:20.953999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.953982    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"
Oct 30 22:30:20.954093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.954038    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv" event={"ID":"5d151563-f8aa-4459-ba60-f048d5978c9a","Type":"ContainerDied","Data":"bde9b2ca51f568531725388053ba0865049037884a091d1479740c44d277dd78"}
Oct 30 22:30:20.955629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.955603    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5" event={"ID":"4970e768-2625-4c3c-ac71-9eeeb2fbe3b5","Type":"ContainerDied","Data":"e6f63bc3588575c5fa2a9de559eb0b1485783104aa2349728708ee29a63e3d5f"}
Oct 30 22:30:20.955748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.955661    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"
Oct 30 22:30:20.975841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:20.975819    5028 scope.go:122] "RemoveContainer" containerID="e601d11c761ace78f633dcf7c040976d2e5b58fc90cb7fc00372beda498a018d"
Oct 30 22:30:21.001000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.000286    5028 scope.go:122] "RemoveContainer" containerID="23828e049de81846168955cfcd83a1741acccac9b02b86ff70ab16c8bdc2a161"
Oct 30 22:30:21.156978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.156896    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:21.235736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.235705    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc\" (UniqueName: \"kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc\") pod \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") "
Oct 30 22:30:21.235998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.235980    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca\") pod \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\" (UID: \"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72\") "
Oct 30 22:30:21.236404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.236381    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca" pod "6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" (UID: "6938ec33-e8f9-4e67-9ee1-284ff6f3fc72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:21.236522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.236508    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.237926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.237899    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc" pod "6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" (UID: "6938ec33-e8f9-4e67-9ee1-284ff6f3fc72"). InnerVolumeSpecName "kube-api-access-mmdgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.279762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.279724    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"]
Oct 30 22:30:21.296587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.296557    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" path="/var/lib/kubelet/pods/37a85e83-9a2b-4994-a152-074b0815e69c/volumes"
Oct 30 22:30:21.297112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.297097    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" path="/var/lib/kubelet/pods/c7c61bab-8baf-4190-ba3c-f291b183dc69/volumes"
Oct 30 22:30:21.297520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.297507    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" path="/var/lib/kubelet/pods/d17f7b66-0a4b-4b50-827f-51b66871f50b/volumes"
Oct 30 22:30:21.298447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.298432    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29889990-4h5xv"]
Oct 30 22:30:21.337974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.337938    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmdgc\" (UniqueName: \"kubernetes.io/projected/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72-kube-api-access-mmdgc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.344732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.344670    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"]
Oct 30 22:30:21.389110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.389077    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"]
Oct 30 22:30:21.394092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.394055    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889990-68nt5"]
Oct 30 22:30:21.966454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.966422    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"
Oct 30 22:30:21.966942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.966426    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr" event={"ID":"6938ec33-e8f9-4e67-9ee1-284ff6f3fc72","Type":"ContainerDied","Data":"8dc1e2a62f64d7c5d6c8229e3f4f7d8efcbc632686f765c0b3203da994958567"}
Oct 30 22:30:21.966942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:21.966587    5028 scope.go:122] "RemoveContainer" containerID="d8a5eed66c599909c9c3e0919dd37c6f4ecb6d22ba5ebe30b8ec6ee105d78f16"
Oct 30 22:30:22.130570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.130536    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"]
Oct 30 22:30:22.135282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.135244    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d2d4c3-29889990-jhvcr"]
Oct 30 22:30:22.166786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.166759    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:22.250518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.250416    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd\" (UniqueName: \"kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd\") pod \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") "
Oct 30 22:30:22.250518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.250470    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca\") pod \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\" (UID: \"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d\") "
Oct 30 22:30:22.250930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.250905    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca" pod "7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" (UID: "7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:22.251234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.251217    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:22.252908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.252878    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd" pod "7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" (UID: "7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d"). InnerVolumeSpecName "kube-api-access-fnmsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:22.352654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.352574    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnmsd\" (UniqueName: \"kubernetes.io/projected/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d-kube-api-access-fnmsd\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:22.982960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.982928    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"
Oct 30 22:30:22.983546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.982964    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc" event={"ID":"7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d","Type":"ContainerDied","Data":"3e5945c2dc2bbda27c4a6852744f3d76317d38b96cbafeeb88ce65a2e8ad92a3"}
Oct 30 22:30:22.983546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:22.983025    5028 scope.go:122] "RemoveContainer" containerID="c0618b519925f7b0148fb0ba07c7c2d64a21b956635855fe28d38e7af35d47df"
Oct 30 22:30:23.144579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:23.144541    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"]
Oct 30 22:30:23.191507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:23.191474    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-af30a9-29889990-lj5tc"]
Oct 30 22:30:23.312944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:23.312843    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" path="/var/lib/kubelet/pods/4970e768-2625-4c3c-ac71-9eeeb2fbe3b5/volumes"
Oct 30 22:30:23.313534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:23.313510    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" path="/var/lib/kubelet/pods/5d151563-f8aa-4459-ba60-f048d5978c9a/volumes"
Oct 30 22:30:23.314149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:23.314128    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" path="/var/lib/kubelet/pods/6938ec33-e8f9-4e67-9ee1-284ff6f3fc72/volumes"
Oct 30 22:30:23.315987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:23.315248    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" path="/var/lib/kubelet/pods/7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d/volumes"
Oct 30 22:30:24.191753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:24.191656    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"]
Oct 30 22:30:24.199257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:24.199217    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181a1b4-xw8hd"]
Oct 30 22:30:25.295310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:25.295281    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d360fdf6-9f19-437f-b963-721f2cc9b087" path="/var/lib/kubelet/pods/d360fdf6-9f19-437f-b963-721f2cc9b087/volumes"
Oct 30 22:30:25.672042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:25.671959    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:28.471233 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:28.471190    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:30:30.153811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:30.153778    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:34.656221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:34.656181    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.358162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:36.358128    5028 generic.go:356] "Generic (PLEG): container finished" podID="5246328d-a9d6-4499-8fd0-c05902265768" containerID="5db3033cf2e5ba970a552d9b29365013c1d6c32e208b523243dc31b6a3cfaedd" exitCode=0
Oct 30 22:30:36.358760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:36.358203    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" event={"ID":"5246328d-a9d6-4499-8fd0-c05902265768","Type":"ContainerDied","Data":"5db3033cf2e5ba970a552d9b29365013c1d6c32e208b523243dc31b6a3cfaedd"}
Oct 30 22:30:36.358760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:36.358682    5028 scope.go:122] "RemoveContainer" containerID="5db3033cf2e5ba970a552d9b29365013c1d6c32e208b523243dc31b6a3cfaedd"
Oct 30 22:30:36.360591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:36.360565    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:30:36.360741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:36.360632    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:30:36.360741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:36.360664    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:30:37.371767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:37.371725    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" event={"ID":"5246328d-a9d6-4499-8fd0-c05902265768","Type":"ContainerStarted","Data":"d92af2841a535b666afbaa6a881fa557c8534f0cf4b20f886b48fd0c45e1540d"}
Oct 30 22:30:38.175902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:38.175860    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:30:38.177600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:38.177577    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:30:38.177700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:38.177626    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:30:38.180553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:38.180531    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:30:39.147753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:39.145503    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:39.179327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:39.176590    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:30:39.179327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:39.176652    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:30:40.176860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:40.176828    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:30:40.177279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:40.176886    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:30:41.177041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:41.177001    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:30:41.177446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:41.177060    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:30:42.176084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:42.176047    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:30:42.176278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:42.176117    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 22:30:43.181417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:43.181384    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:30:43.182759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:43.182733    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:30:43.848422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:43.848389    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:48.784020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:48.783978    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.098747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:49.098665    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:50.125429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.125389    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-c03ad2"]
Oct 30 22:30:50.126228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126210    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126230    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126245    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126250    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126258    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126263    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126270    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.126274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126275    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126282    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126286    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126295    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126299    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126307    5028 cpu_manager.go:414] "containerMap: removing container" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126311    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126319    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126323    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126338    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126342    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126351    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126355    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126362    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126367    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126373    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126377    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126385    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126389    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126395    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126398    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126414    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126419    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126426    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126430    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126437    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126441    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126449    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126453    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126466    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126471    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126478    5028 cpu_manager.go:414] "containerMap: removing container" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126483    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126490    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126494    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126502    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:50.126504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126506    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126605    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126614    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126619    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126625    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126630    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126635    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126641    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126648    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126655    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126661    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf357047-0d19-46fd-a69e-0c62b92039c6" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126665    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126671    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126676    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126682    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126704    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126710    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126717    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4970e768-2625-4c3c-ac71-9eeeb2fbe3b5" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126723    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126729    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126735    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:50.128139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.126741    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:50.129738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.129714    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:50.182909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.182875    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9v4z\""
Oct 30 22:30:50.188639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.188607    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-c03ad2"]
Oct 30 22:30:50.208299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.208261    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8774\" (UniqueName: \"kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774\") pod \"test-curl-7123-c03ad2\" (UID: \"b39eac83-ec61-4e37-8d07-d8c199510dde\") " pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:50.310398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.310355    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8774\" (UniqueName: \"kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774\") pod \"test-curl-7123-c03ad2\" (UID: \"b39eac83-ec61-4e37-8d07-d8c199510dde\") " pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:50.369707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.369664    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8774\" (UniqueName: \"kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774\") pod \"test-curl-7123-c03ad2\" (UID: \"b39eac83-ec61-4e37-8d07-d8c199510dde\") " pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:50.442745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.442643    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:50.818661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:50.818632    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-c03ad2"]
Oct 30 22:30:51.506295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:51.506249    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-c03ad2" event={"ID":"b39eac83-ec61-4e37-8d07-d8c199510dde","Type":"ContainerStarted","Data":"7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a"}
Oct 30 22:30:51.506761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:51.506301    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-c03ad2" event={"ID":"b39eac83-ec61-4e37-8d07-d8c199510dde","Type":"ContainerStarted","Data":"693fc0d1eab432cd3f635427793f6b10fe5ba67ecb2aab8745953b517b95baac"}
Oct 30 22:30:51.617022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:51.616972    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-c03ad2" podStartSLOduration=1.616958685 podStartE2EDuration="1.616958685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:51.614155614 +0000 UTC m=+1782.463668584" watchObservedRunningTime="2026-10-30 22:30:51.616958685 +0000 UTC m=+1782.466471657"
Oct 30 22:30:53.206916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.206881    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-c03ad2"]
Oct 30 22:30:53.211438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.211409    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-c03ad2"]
Oct 30 22:30:53.292954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.292908    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.294667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.294646    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-08b284"]
Oct 30 22:30:53.295454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295437    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:53.295503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295456    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:53.295503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295474    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:53.295503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295479    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295508    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295514    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295535    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295540    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c61bab-8baf-4190-ba3c-f291b183dc69" containerName="cleanup"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295553    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" containerName="test-curl-7123-c03ad2"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295559    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" containerName="test-curl-7123-c03ad2"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295568    5028 cpu_manager.go:414] "containerMap: removing container" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:53.295601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295575    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295714    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf83d7a-15cb-4287-85a6-0f6b4e28dc4d" containerName="cleanup"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295730    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" containerName="test-curl-7123-c03ad2"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295742    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d151563-f8aa-4459-ba60-f048d5978c9a" containerName="cleanup"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295751    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5e07d1-d36a-447d-906b-6eec1de1cd44" containerName="cleanup"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295761    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17f7b66-0a4b-4b50-827f-51b66871f50b" containerName="cleanup"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295769    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6938ec33-e8f9-4e67-9ee1-284ff6f3fc72" containerName="cleanup"
Oct 30 22:30:53.295966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.295777    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a85e83-9a2b-4994-a152-074b0815e69c" containerName="cleanup"
Oct 30 22:30:53.298508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.298492    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:53.373554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.366834    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdsc\" (UniqueName: \"kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc\") pod \"test-curl-7123-08b284\" (UID: \"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e\") " pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:53.373554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.372739    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-08b284"]
Oct 30 22:30:53.469011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.468971    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdsc\" (UniqueName: \"kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc\") pod \"test-curl-7123-08b284\" (UID: \"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e\") " pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:53.523571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.523531    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-c03ad2" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" containerName="test-curl-7123-c03ad2" containerID="cri-o://7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a" gracePeriod=2
Oct 30 22:30:53.528065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.528040    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdsc\" (UniqueName: \"kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc\") pod \"test-curl-7123-08b284\" (UID: \"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e\") " pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:53.610943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.610914    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:53.991055 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:53.991016    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa26221_764b_4c6c_9ba2_8c8e3db7ab9e.slice/crio-c05c5796fa719854eb2e59d1c7da635e25f929b2bbf3ded7863307e29e55d209 WatchSource:0}: Error finding container c05c5796fa719854eb2e59d1c7da635e25f929b2bbf3ded7863307e29e55d209: Status 404 returned error can't find the container with id c05c5796fa719854eb2e59d1c7da635e25f929b2bbf3ded7863307e29e55d209
Oct 30 22:30:53.991680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:53.991648    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-08b284"]
Oct 30 22:30:54.533769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:54.533732    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-08b284" event={"ID":"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e","Type":"ContainerStarted","Data":"1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399"}
Oct 30 22:30:54.534164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:54.533778    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-08b284" event={"ID":"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e","Type":"ContainerStarted","Data":"c05c5796fa719854eb2e59d1c7da635e25f929b2bbf3ded7863307e29e55d209"}
Oct 30 22:30:54.647112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:54.647057    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-08b284" podStartSLOduration=1.647043328 podStartE2EDuration="1.647043328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:54.642657396 +0000 UTC m=+1785.492170370" watchObservedRunningTime="2026-10-30 22:30:54.647043328 +0000 UTC m=+1785.496556301"
Oct 30 22:30:55.792350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:55.792287    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:55.907507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:55.907475    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774\" (UniqueName: \"kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774\") pod \"b39eac83-ec61-4e37-8d07-d8c199510dde\" (UID: \"b39eac83-ec61-4e37-8d07-d8c199510dde\") "
Oct 30 22:30:55.909533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:55.909502    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774" pod "b39eac83-ec61-4e37-8d07-d8c199510dde" (UID: "b39eac83-ec61-4e37-8d07-d8c199510dde"). InnerVolumeSpecName "kube-api-access-q8774". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:56.009936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.009902    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8774\" (UniqueName: \"kubernetes.io/projected/b39eac83-ec61-4e37-8d07-d8c199510dde-kube-api-access-q8774\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:56.247719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.247670    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-08b284"]
Oct 30 22:30:56.256763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.256716    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-08b284"]
Oct 30 22:30:56.417011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.416973    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-7b7995"]
Oct 30 22:30:56.418188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.418165    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" containerName="test-curl-7123-08b284"
Oct 30 22:30:56.418288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.418191    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" containerName="test-curl-7123-08b284"
Oct 30 22:30:56.418423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.418412    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" containerName="test-curl-7123-08b284"
Oct 30 22:30:56.422150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.422128    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:30:56.475881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.475848    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-7b7995"]
Oct 30 22:30:56.518405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.518321    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndkg\" (UniqueName: \"kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg\") pod \"test-curl-7123-7b7995\" (UID: \"be35f36c-bb60-480e-96d9-55d481e46071\") " pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:30:56.570246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.570211    5028 generic.go:356] "Generic (PLEG): container finished" podID="b39eac83-ec61-4e37-8d07-d8c199510dde" containerID="7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a" exitCode=137
Oct 30 22:30:56.570420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.570266    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:56.570420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.570305    5028 scope.go:122] "RemoveContainer" containerID="7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a"
Oct 30 22:30:56.570539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.570457    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-08b284" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" containerName="test-curl-7123-08b284" containerID="cri-o://1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399" gracePeriod=2
Oct 30 22:30:56.581735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.581680    5028 scope.go:122] "RemoveContainer" containerID="7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a"
Oct 30 22:30:56.582014 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:56.581996    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a\": container with ID starting with 7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a not found: ID does not exist" containerID="7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a"
Oct 30 22:30:56.582087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.582027    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a"} err="failed to get container status \"7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a\": rpc error: code = NotFound desc = could not find container \"7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a\": container with ID starting with 7bc80e746cb3089456cbb5f6ed97a275d9207742b4fd01e306d99b95e8697d3a not found: ID does not exist"
Oct 30 22:30:56.619318 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:56.619272    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-08b284\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:56.620029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.620000    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndkg\" (UniqueName: \"kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg\") pod \"test-curl-7123-7b7995\" (UID: \"be35f36c-bb60-480e-96d9-55d481e46071\") " pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:30:56.668250 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:56.668207    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-08b284\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:56.679124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.679094    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndkg\" (UniqueName: \"kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg\") pod \"test-curl-7123-7b7995\" (UID: \"be35f36c-bb60-480e-96d9-55d481e46071\") " pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:30:56.716816 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:56.716775    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-c03ad2\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:56.737822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.737779    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:30:56.871696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.871647    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"]
Oct 30 22:30:56.878797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.878759    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:56.927602 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:56.927568    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-08b284\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:56.929787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.929735    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:56.929787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.929753    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:56.929999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.929842    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:56.938106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.937354    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"]
Oct 30 22:30:56.969722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:56.969665    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:30:56.976029 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:56.975998    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-c03ad2\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" pod="opendatahub/test-curl-7123-c03ad2"
Oct 30 22:30:57.027517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.027481    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.027756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.027613    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgpl\" (UniqueName: \"kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.027756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.027743    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.115423 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:57.115376    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe35f36c_bb60_480e_96d9_55d481e46071.slice/crio-29dabef653a3fae2ebe48cf662866abc32cfca299c20a4b1eadc9ebe551fbeb3 WatchSource:0}: Error finding container 29dabef653a3fae2ebe48cf662866abc32cfca299c20a4b1eadc9ebe551fbeb3: Status 404 returned error can't find the container with id 29dabef653a3fae2ebe48cf662866abc32cfca299c20a4b1eadc9ebe551fbeb3
Oct 30 22:30:57.115747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.115721    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-7b7995"]
Oct 30 22:30:57.129458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.129426    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.129576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.129538    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgpl\" (UniqueName: \"kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.129697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.129668    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.130239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.130208    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.132073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.132051    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.188654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.188627    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgpl\" (UniqueName: \"kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl\") pod \"maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.195279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.195256    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:30:57.300505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.300470    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39eac83-ec61-4e37-8d07-d8c199510dde" path="/var/lib/kubelet/pods/b39eac83-ec61-4e37-8d07-d8c199510dde/volumes"
Oct 30 22:30:57.577298 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:30:57.577260    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f484b5_807c_466f_9606_af960b86b1f9.slice/crio-d0d8c861eec84d4683f406a8bfc2c05e10b2a3a2693860974bb530edb63e9913 WatchSource:0}: Error finding container d0d8c861eec84d4683f406a8bfc2c05e10b2a3a2693860974bb530edb63e9913: Status 404 returned error can't find the container with id d0d8c861eec84d4683f406a8bfc2c05e10b2a3a2693860974bb530edb63e9913
Oct 30 22:30:57.579936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.579880    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"]
Oct 30 22:30:57.583557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.583529    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-7b7995" event={"ID":"be35f36c-bb60-480e-96d9-55d481e46071","Type":"ContainerStarted","Data":"7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023"}
Oct 30 22:30:57.583683 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.583569    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-7b7995" event={"ID":"be35f36c-bb60-480e-96d9-55d481e46071","Type":"ContainerStarted","Data":"29dabef653a3fae2ebe48cf662866abc32cfca299c20a4b1eadc9ebe551fbeb3"}
Oct 30 22:30:57.697208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.697146    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-7b7995" podStartSLOduration=1.6971289029999999 podStartE2EDuration="1.697128903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:57.692392212 +0000 UTC m=+1788.541905188" watchObservedRunningTime="2026-10-30 22:30:57.697128903 +0000 UTC m=+1788.546641879"
Oct 30 22:30:57.955025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:57.954986    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:58.441141 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:58.441091    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:30:58.596958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:58.596922    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj" event={"ID":"02f484b5-807c-466f-9606-af960b86b1f9","Type":"ContainerStarted","Data":"0e3fa3ec2fba96e08869f4fd0188a597ea2c1fa1181938ca4f3db3d0d791ab75"}
Oct 30 22:30:58.597089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:58.596985    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj" event={"ID":"02f484b5-807c-466f-9606-af960b86b1f9","Type":"ContainerStarted","Data":"d0d8c861eec84d4683f406a8bfc2c05e10b2a3a2693860974bb530edb63e9913"}
Oct 30 22:30:58.714168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:58.714115    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj" podStartSLOduration=2.714101878 podStartE2EDuration="2.714101878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.711030444 +0000 UTC m=+1789.560543418" watchObservedRunningTime="2026-10-30 22:30:58.714101878 +0000 UTC m=+1789.563614864"
Oct 30 22:30:58.862470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:58.862439    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:58.958271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:58.958238    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc\" (UniqueName: \"kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc\") pod \"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e\" (UID: \"5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e\") "
Oct 30 22:30:58.960982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:58.960940    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc" pod "5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" (UID: "5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e"). InnerVolumeSpecName "kube-api-access-6kdsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:59.060843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.060802    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kdsc\" (UniqueName: \"kubernetes.io/projected/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e-kube-api-access-6kdsc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:59.179065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.179026    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-7b7995"]
Oct 30 22:30:59.183764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.183730    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-7b7995"]
Oct 30 22:30:59.294568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.294491    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" path="/var/lib/kubelet/pods/5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e/volumes"
Oct 30 22:30:59.341042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.341010    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-81d28f"]
Oct 30 22:30:59.342209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.342189    5028 cpu_manager.go:414] "containerMap: removing container" podUID="be35f36c-bb60-480e-96d9-55d481e46071" containerName="test-curl-7123-7b7995"
Oct 30 22:30:59.342296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.342210    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be35f36c-bb60-480e-96d9-55d481e46071" containerName="test-curl-7123-7b7995"
Oct 30 22:30:59.342460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.342448    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="be35f36c-bb60-480e-96d9-55d481e46071" containerName="test-curl-7123-7b7995"
Oct 30 22:30:59.345990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.345971    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:30:59.398012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.397977    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-81d28f"]
Oct 30 22:30:59.468047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.468004    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kds8l\" (UniqueName: \"kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l\") pod \"test-curl-7123-81d28f\" (UID: \"f4486117-dc53-45c8-aeab-ef24d75d81c4\") " pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:30:59.569998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.569900    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kds8l\" (UniqueName: \"kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l\") pod \"test-curl-7123-81d28f\" (UID: \"f4486117-dc53-45c8-aeab-ef24d75d81c4\") " pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:30:59.607275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.607242    5028 generic.go:356] "Generic (PLEG): container finished" podID="5fa26221-764b-4c6c-9ba2-8c8e3db7ab9e" containerID="1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399" exitCode=137
Oct 30 22:30:59.607464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.607291    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-08b284"
Oct 30 22:30:59.607464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.607353    5028 scope.go:122] "RemoveContainer" containerID="1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399"
Oct 30 22:30:59.621855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.621790    5028 generic.go:356] "Generic (PLEG): container finished" podID="02f484b5-807c-466f-9606-af960b86b1f9" containerID="0e3fa3ec2fba96e08869f4fd0188a597ea2c1fa1181938ca4f3db3d0d791ab75" exitCode=0
Oct 30 22:30:59.622207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.621926    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj" event={"ID":"02f484b5-807c-466f-9606-af960b86b1f9","Type":"ContainerDied","Data":"0e3fa3ec2fba96e08869f4fd0188a597ea2c1fa1181938ca4f3db3d0d791ab75"}
Oct 30 22:30:59.622403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.622381    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-7b7995" podUID="be35f36c-bb60-480e-96d9-55d481e46071" containerName="test-curl-7123-7b7995" containerID="cri-o://7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023" gracePeriod=2
Oct 30 22:30:59.628536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.628511    5028 scope.go:122] "RemoveContainer" containerID="1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399"
Oct 30 22:30:59.628649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.628625    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kds8l\" (UniqueName: \"kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l\") pod \"test-curl-7123-81d28f\" (UID: \"f4486117-dc53-45c8-aeab-ef24d75d81c4\") " pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:30:59.629034 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:30:59.629011    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399\": container with ID starting with 1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399 not found: ID does not exist" containerID="1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399"
Oct 30 22:30:59.629133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.629039    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399"} err="failed to get container status \"1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399\": rpc error: code = NotFound desc = could not find container \"1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399\": container with ID starting with 1ec8082a83c0de0c12794fb7a96c5a93db587bd254ae623fd305a12ab2bf2399 not found: ID does not exist"
Oct 30 22:30:59.658963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:30:59.658929    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:31:00.036405 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:31:00.036363    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4486117_dc53_45c8_aeab_ef24d75d81c4.slice/crio-5493ba583541914a1feb1a7220c223c1af8902a1a529e112e4bc6a25034aa6e8 WatchSource:0}: Error finding container 5493ba583541914a1feb1a7220c223c1af8902a1a529e112e4bc6a25034aa6e8: Status 404 returned error can't find the container with id 5493ba583541914a1feb1a7220c223c1af8902a1a529e112e4bc6a25034aa6e8
Oct 30 22:31:00.036879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.036783    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-81d28f"]
Oct 30 22:31:00.635774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.635736    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-81d28f" event={"ID":"f4486117-dc53-45c8-aeab-ef24d75d81c4","Type":"ContainerStarted","Data":"47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779"}
Oct 30 22:31:00.635937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.635785    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-81d28f" event={"ID":"f4486117-dc53-45c8-aeab-ef24d75d81c4","Type":"ContainerStarted","Data":"5493ba583541914a1feb1a7220c223c1af8902a1a529e112e4bc6a25034aa6e8"}
Oct 30 22:31:00.749486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.749363    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-81d28f" podStartSLOduration=1.749342807 podStartE2EDuration="1.749342807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:00.746738308 +0000 UTC m=+1791.596251278" watchObservedRunningTime="2026-10-30 22:31:00.749342807 +0000 UTC m=+1791.598855781"
Oct 30 22:31:00.809867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.809837    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:31:00.990015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.989982    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl\" (UniqueName: \"kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl\") pod \"02f484b5-807c-466f-9606-af960b86b1f9\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") "
Oct 30 22:31:00.990982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.990136    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca\") pod \"02f484b5-807c-466f-9606-af960b86b1f9\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") "
Oct 30 22:31:00.990982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.990211    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca\") pod \"02f484b5-807c-466f-9606-af960b86b1f9\" (UID: \"02f484b5-807c-466f-9606-af960b86b1f9\") "
Oct 30 22:31:00.990982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.990607    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca" pod "02f484b5-807c-466f-9606-af960b86b1f9" (UID: "02f484b5-807c-466f-9606-af960b86b1f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:00.990982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.990887    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.992502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.992478    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl" pod "02f484b5-807c-466f-9606-af960b86b1f9" (UID: "02f484b5-807c-466f-9606-af960b86b1f9"). InnerVolumeSpecName "kube-api-access-tzgpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:00.992815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:00.992794    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca" pod "02f484b5-807c-466f-9606-af960b86b1f9" (UID: "02f484b5-807c-466f-9606-af960b86b1f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:01.092116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:01.092078    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgpl\" (UniqueName: \"kubernetes.io/projected/02f484b5-807c-466f-9606-af960b86b1f9-kube-api-access-tzgpl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.092116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:01.092109    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f484b5-807c-466f-9606-af960b86b1f9-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.649221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:01.649196    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"
Oct 30 22:31:01.649489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:01.649227    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj" event={"ID":"02f484b5-807c-466f-9606-af960b86b1f9","Type":"ContainerDied","Data":"d0d8c861eec84d4683f406a8bfc2c05e10b2a3a2693860974bb530edb63e9913"}
Oct 30 22:31:01.649604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:01.649512    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d8c861eec84d4683f406a8bfc2c05e10b2a3a2693860974bb530edb63e9913"
Oct 30 22:31:01.948349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:01.948320    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:31:02.004148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.004114    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg\" (UniqueName: \"kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg\") pod \"be35f36c-bb60-480e-96d9-55d481e46071\" (UID: \"be35f36c-bb60-480e-96d9-55d481e46071\") "
Oct 30 22:31:02.006172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.006146    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg" pod "be35f36c-bb60-480e-96d9-55d481e46071" (UID: "be35f36c-bb60-480e-96d9-55d481e46071"). InnerVolumeSpecName "kube-api-access-zndkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:02.105740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.105699    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndkg\" (UniqueName: \"kubernetes.io/projected/be35f36c-bb60-480e-96d9-55d481e46071-kube-api-access-zndkg\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:02.344951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.344912    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-81d28f"]
Oct 30 22:31:02.350960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.350931    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-81d28f"]
Oct 30 22:31:02.662427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.662328    5028 generic.go:356] "Generic (PLEG): container finished" podID="be35f36c-bb60-480e-96d9-55d481e46071" containerID="7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023" exitCode=137
Oct 30 22:31:02.662427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.662372    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:31:02.662427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.662381    5028 scope.go:122] "RemoveContainer" containerID="7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023"
Oct 30 22:31:02.662802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.662778    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-81d28f" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" containerName="test-curl-7123-81d28f" containerID="cri-o://47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779" gracePeriod=2
Oct 30 22:31:02.678258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.678238    5028 scope.go:122] "RemoveContainer" containerID="7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023"
Oct 30 22:31:02.678549 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:02.678527    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023\": container with ID starting with 7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023 not found: ID does not exist" containerID="7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023"
Oct 30 22:31:02.678615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.678559    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023"} err="failed to get container status \"7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023\": rpc error: code = NotFound desc = could not find container \"7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023\": container with ID starting with 7bc671e870edb3d668c8e1ec165e19048e4faff2522f7dcc6f9c75cc5163b023 not found: ID does not exist"
Oct 30 22:31:02.711759 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:02.711727    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-81d28f\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:31:02.760545 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:02.760505    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-81d28f\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:31:02.808963 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:02.808929    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-7b7995\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="be35f36c-bb60-480e-96d9-55d481e46071" pod="opendatahub/test-curl-7123-7b7995"
Oct 30 22:31:02.901945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:02.901873    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:03.295540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:03.295504    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be35f36c-bb60-480e-96d9-55d481e46071" path="/var/lib/kubelet/pods/be35f36c-bb60-480e-96d9-55d481e46071/volumes"
Oct 30 22:31:03.424953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:03.424920    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.731700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.731641    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"]
Oct 30 22:31:04.732943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.732901    5028 cpu_manager.go:414] "containerMap: removing container" podUID="02f484b5-807c-466f-9606-af960b86b1f9" containerName="revoke-keys"
Oct 30 22:31:04.732943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.732928    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f484b5-807c-466f-9606-af960b86b1f9" containerName="revoke-keys"
Oct 30 22:31:04.733174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.733002    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" containerName="test-curl-7123-81d28f"
Oct 30 22:31:04.733174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.733011    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" containerName="test-curl-7123-81d28f"
Oct 30 22:31:04.733292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.733185    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" containerName="test-curl-7123-81d28f"
Oct 30 22:31:04.733292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.733208    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f484b5-807c-466f-9606-af960b86b1f9" containerName="revoke-keys"
Oct 30 22:31:04.740377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.737112    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755488    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755578    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755620    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755676    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755777    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755856    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.755912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.755892    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrpf\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.756170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.756073    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.756229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.756214    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.791603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.791522    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-721c75-openshift-default-dockercfg-srq8s\""
Oct 30 22:31:04.850394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.850357    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"]
Oct 30 22:31:04.851898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.851870    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.858193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.858154    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.860835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.860802    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrpf\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.861180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.861160    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.861385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.861370    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.861538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.861525    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.861713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.861681    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.861844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.861830    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.861986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.861971    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.862180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.862163    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.862944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.862923    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.864281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.864258    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.870746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.867802    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.870746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.868373    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.870746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.869071    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.871039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.871016    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.871560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.871538    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.928795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.928672    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrpf\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:04.930586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:04.930559    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token\") pod \"e2e-api-a-721c75-openshift-default-74d994d999-g9nsl\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") " pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:05.007727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.007119    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:31:05.043710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.039763    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:05.063810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.063763    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:05.077355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.077314    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l\" (UniqueName: \"kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l\") pod \"f4486117-dc53-45c8-aeab-ef24d75d81c4\" (UID: \"f4486117-dc53-45c8-aeab-ef24d75d81c4\") "
Oct 30 22:31:05.080631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.080598    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l" pod "f4486117-dc53-45c8-aeab-ef24d75d81c4" (UID: "f4486117-dc53-45c8-aeab-ef24d75d81c4"). InnerVolumeSpecName "kube-api-access-kds8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:05.179329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.179252    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kds8l\" (UniqueName: \"kubernetes.io/projected/f4486117-dc53-45c8-aeab-ef24d75d81c4-kube-api-access-kds8l\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:05.294313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.294280    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4486117-dc53-45c8-aeab-ef24d75d81c4" path="/var/lib/kubelet/pods/f4486117-dc53-45c8-aeab-ef24d75d81c4/volumes"
Oct 30 22:31:05.500592 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:31:05.500558    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3f5d9c_da8a_48b1_a048_17aa76541c4d.slice/crio-e4196a1e32e099449a5808adbfb952245800ab847d10f02635a4f4d4abd7f2e9 WatchSource:0}: Error finding container e4196a1e32e099449a5808adbfb952245800ab847d10f02635a4f4d4abd7f2e9: Status 404 returned error can't find the container with id e4196a1e32e099449a5808adbfb952245800ab847d10f02635a4f4d4abd7f2e9
Oct 30 22:31:05.501188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.501164    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"]
Oct 30 22:31:05.503464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.503422    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:05.503575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.503498    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:05.503575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.503533    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:05.705670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.705638    5028 generic.go:356] "Generic (PLEG): container finished" podID="f4486117-dc53-45c8-aeab-ef24d75d81c4" containerID="47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779" exitCode=137
Oct 30 22:31:05.705913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.705680    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-81d28f"
Oct 30 22:31:05.705913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.705738    5028 scope.go:122] "RemoveContainer" containerID="47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779"
Oct 30 22:31:05.708071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.708007    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" event={"ID":"1a3f5d9c-da8a-48b1-a048-17aa76541c4d","Type":"ContainerStarted","Data":"c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"}
Oct 30 22:31:05.708193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.708085    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" event={"ID":"1a3f5d9c-da8a-48b1-a048-17aa76541c4d","Type":"ContainerStarted","Data":"e4196a1e32e099449a5808adbfb952245800ab847d10f02635a4f4d4abd7f2e9"}
Oct 30 22:31:05.717903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.717867    5028 scope.go:122] "RemoveContainer" containerID="47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779"
Oct 30 22:31:05.718225 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:05.718205    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779\": container with ID starting with 47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779 not found: ID does not exist" containerID="47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779"
Oct 30 22:31:05.718310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.718234    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779"} err="failed to get container status \"47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779\": rpc error: code = NotFound desc = could not find container \"47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779\": container with ID starting with 47ecf63b7764d5611f42ddac7fd5cdece6954db8c217fdf9285c1b0ec16b1779 not found: ID does not exist"
Oct 30 22:31:05.851410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:05.851279    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podStartSLOduration=1.8512570560000001 podStartE2EDuration="1.851257056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:05.848523251 +0000 UTC m=+1796.698036225" watchObservedRunningTime="2026-10-30 22:31:05.851257056 +0000 UTC m=+1796.700770030"
Oct 30 22:31:06.064039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:06.063992    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:06.071081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:06.071049    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:06.719762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:06.719726    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:06.720942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:06.720916    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:31:07.871873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:07.871429    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:08.700402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:08.700357    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:08.928865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:08.928817    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:09.136646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:09.136589    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:09.874910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:09.874855    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:10.088834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:10.088800    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:31:10.123324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:10.123293    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:31:12.599905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:12.599837    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:12.846731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:12.834109    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:13.968163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:13.968124    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:14.148117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.094741    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"]
Oct 30 22:31:14.298768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.287952    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"]
Oct 30 22:31:14.298768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.288008    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"]
Oct 30 22:31:14.345477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.324827    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.349807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.349775    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"]
Oct 30 22:31:14.350390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.350368    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.378743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.377470    5028 scope.go:122] "RemoveContainer" containerID="5f6c079f80998556a23e26849fb4dd8bcf744d3d508613db6f9a2289f1e21f25"
Oct 30 22:31:14.400541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.399964    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7lxg\" (UniqueName: \"kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg\") pod \"payload-processing-e2e-api-a-721c75-6b765575f7-zklj8\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") " pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.400541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.400115    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-721c75-6b765575f7-zklj8\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") " pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.442630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.442593    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-721c75-dockercfg-kz4b8\""
Oct 30 22:31:14.443057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.443035    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-721c75\""
Oct 30 22:31:14.504625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.504568    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mz5m\" (UniqueName: \"kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m\") pod \"payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.505259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.505211    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7lxg\" (UniqueName: \"kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg\") pod \"payload-processing-e2e-api-a-721c75-6b765575f7-zklj8\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") " pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.505480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.505458    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.505626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.505607    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-721c75-6b765575f7-zklj8\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") " pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.506881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.506861    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-721c75-6b765575f7-zklj8\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") " pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.604311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.604281    5028 scope.go:122] "RemoveContainer" containerID="76f587fbcc6ba0a4cea9219f470552243f70383e0dd09e59034f53c1c56aea55"
Oct 30 22:31:14.626889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.613305    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mz5m\" (UniqueName: \"kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m\") pod \"payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.627558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.627531    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.628797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.628769    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.628946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.613886    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7lxg\" (UniqueName: \"kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg\") pod \"payload-processing-e2e-api-a-721c75-6b765575f7-zklj8\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") " pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.737313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.737270    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mz5m\" (UniqueName: \"kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m\") pod \"payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:14.766784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.765674    5028 scope.go:122] "RemoveContainer" containerID="6a73b49e0a1a1af4f636c6d64b31f803711eedf51b6a8a7cd91120c7f6ce8943"
Oct 30 22:31:14.912328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.912291    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:14.964713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:14.963726    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:16.227167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:16.227097    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:16.372070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:16.364820    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:31:16.404706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:16.394798    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:16.404706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:16.399890    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:16.431709 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:16.431269    5028 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.121s"
Oct 30 22:31:16.431709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:16.431318    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"]
Oct 30 22:31:17.157436 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:31:17.157402    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828c50d6_ff1a_488a_a07d_f778706b5027.slice/crio-a6fc2670db92727646b6d047c3225d87437db1edc74d082ffa97bd496d55d98b WatchSource:0}: Error finding container a6fc2670db92727646b6d047c3225d87437db1edc74d082ffa97bd496d55d98b: Status 404 returned error can't find the container with id a6fc2670db92727646b6d047c3225d87437db1edc74d082ffa97bd496d55d98b
Oct 30 22:31:17.163301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.163274    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"]
Oct 30 22:31:17.478922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.478857    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7" event={"ID":"828c50d6-ff1a-488a-a07d-f778706b5027","Type":"ContainerStarted","Data":"ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e"}
Oct 30 22:31:17.478922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.478926    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7" event={"ID":"828c50d6-ff1a-488a-a07d-f778706b5027","Type":"ContainerStarted","Data":"a6fc2670db92727646b6d047c3225d87437db1edc74d082ffa97bd496d55d98b"}
Oct 30 22:31:17.479471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.478953    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:17.481266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.481236    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8" event={"ID":"7a931850-938f-469b-9361-c81359628ee2","Type":"ContainerStarted","Data":"9a64dfde34945437672b9759e6c10e44a288b319f64fc473cf22b8ed4f133f3b"}
Oct 30 22:31:17.481431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.481281    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8" event={"ID":"7a931850-938f-469b-9361-c81359628ee2","Type":"ContainerStarted","Data":"141c96718973d1efda4986fafe92bbae1821fa4a4ed4fc4ab0411601d211414b"}
Oct 30 22:31:17.481431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.481318    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:17.606357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.606287    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7" podStartSLOduration=4.606267905 podStartE2EDuration="4.606267905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:17.603328363 +0000 UTC m=+1808.452841334" watchObservedRunningTime="2026-10-30 22:31:17.606267905 +0000 UTC m=+1808.455780878"
Oct 30 22:31:17.737819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:17.736943    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8" podStartSLOduration=4.7369226300000005 podStartE2EDuration="4.73692263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:17.736238295 +0000 UTC m=+1808.585751262" watchObservedRunningTime="2026-10-30 22:31:17.73692263 +0000 UTC m=+1808.586435603"
Oct 30 22:31:18.809804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:18.809707    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:18.998279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:18.997512    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:20.486376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:20.486322    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.032413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.032377    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"]
Oct 30 22:31:21.123716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.118798    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.261728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.261670    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"]
Oct 30 22:31:21.277064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.276996    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277191    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277223    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277327    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277389    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277514    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277583    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277644    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.277758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.277734    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdhp\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.291268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.291139    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c3c2cf-openshift-default-dockercfg-52dd4\""
Oct 30 22:31:21.336043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.326271    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.397458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.397425    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.401862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.401677    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.401862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.401815    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.402240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.401872    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.402240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.402064    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.402240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.402123    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.402240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.402175    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.402240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.402231    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdhp\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.402513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.402277    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.403210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.402959    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.404719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.403676    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.405147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.405104    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.409902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.409870    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.410403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.410213    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.410524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.410457    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.410841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.410723    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.465912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.465839    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.466922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.466859    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdhp\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp\") pod \"e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") " pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:21.499489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.499410    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:21.619972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.619875    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:31:21.619972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.619960    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:21.699730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:21.699679    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:22.230671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.230644    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"]
Oct 30 22:31:22.233489 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:31:22.233452    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbea46f_84fc_4ea1_9c67_e395f6bc0748.slice/crio-9152dfeb3045a76f9929f6aa529f2acba4ba72e28f696ac00827f2e1a5edeb15 WatchSource:0}: Error finding container 9152dfeb3045a76f9929f6aa529f2acba4ba72e28f696ac00827f2e1a5edeb15: Status 404 returned error can't find the container with id 9152dfeb3045a76f9929f6aa529f2acba4ba72e28f696ac00827f2e1a5edeb15
Oct 30 22:31:22.236492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.236452    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:22.236609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.236524    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:22.236609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.236553    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:22.552150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.552062    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" event={"ID":"bcbea46f-84fc-4ea1-9c67-e395f6bc0748","Type":"ContainerStarted","Data":"e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"}
Oct 30 22:31:22.552150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.552124    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" event={"ID":"bcbea46f-84fc-4ea1-9c67-e395f6bc0748","Type":"ContainerStarted","Data":"9152dfeb3045a76f9929f6aa529f2acba4ba72e28f696ac00827f2e1a5edeb15"}
Oct 30 22:31:22.668494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.668434    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podStartSLOduration=2.668417255 podStartE2EDuration="2.668417255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:22.667222624 +0000 UTC m=+1813.516735598" watchObservedRunningTime="2026-10-30 22:31:22.668417255 +0000 UTC m=+1813.517930230"
Oct 30 22:31:22.700050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:22.700003    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:23.493727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:23.492922    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"]
Oct 30 22:31:23.493727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:23.493135    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy" containerID="cri-o://d0be13a468bec3d5782a4e5b943c01485ea4bbadba947a8c82952e8970550970" gracePeriod=30
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:23.682646    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:23.706372    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:23.914806    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:23.914920    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:24.142857    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"]
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:24.143043    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy" containerID="cri-o://755e196a5498dd8ceaff6e58ea50e72febb305af7e5a9a58cfa300f9311c28e6" gracePeriod=30
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:24.585780    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:25.326077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:24.587493    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:31:25.466078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.466039    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.597539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.597429    5028 generic.go:356] "Generic (PLEG): container finished" podID="256a1079-f087-481a-98ad-745f95415d8f" containerID="9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97" exitCode=0
Oct 30 22:31:25.597539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.597504    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" event={"ID":"256a1079-f087-481a-98ad-745f95415d8f","Type":"ContainerDied","Data":"9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"}
Oct 30 22:31:25.598083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.598064    5028 scope.go:122] "RemoveContainer" containerID="9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"
Oct 30 22:31:25.600022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.600000    5028 generic.go:356] "Generic (PLEG): container finished" podID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerID="99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5" exitCode=0
Oct 30 22:31:25.600250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.600228    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" event={"ID":"6ed80fe5-2bbd-4988-a7d9-da7026e965c0","Type":"ContainerDied","Data":"99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"}
Oct 30 22:31:25.600660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.600619    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:25.600794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.600730    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:25.600794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.600780    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:25.600907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.600871    5028 scope.go:122] "RemoveContainer" containerID="99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"
Oct 30 22:31:25.602540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.602511    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:25.602647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.602587    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:25.602647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.602636    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:25.668977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.668940    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.857618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:25.857495    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:26.613214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:26.613180    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" event={"ID":"256a1079-f087-481a-98ad-745f95415d8f","Type":"ContainerStarted","Data":"478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31"}
Oct 30 22:31:26.615531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:26.615505    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" event={"ID":"6ed80fe5-2bbd-4988-a7d9-da7026e965c0","Type":"ContainerStarted","Data":"fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec"}
Oct 30 22:31:26.693101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:26.693066    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:31:26.693337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:26.693310    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:26.693401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:26.693382    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:27.290544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.290501    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:27.290772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.290578    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 22:31:27.300092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.300049    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:31:27.669471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.669366    5028 generic.go:356] "Generic (PLEG): container finished" podID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerID="7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac" exitCode=0
Oct 30 22:31:27.669997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.669912    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" event={"ID":"f6de087b-d027-4356-a2a2-4e55b5a258cf","Type":"ContainerDied","Data":"7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"}
Oct 30 22:31:27.670389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.670354    5028 scope.go:122] "RemoveContainer" containerID="7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"
Oct 30 22:31:27.672570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.672527    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:27.672738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.672621    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:27.672738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.672668    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:27.693672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.693650    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:27.693761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:27.693725    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:28.265713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.263100    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:31:28.296711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.294160    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:28.296711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.294222    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 22:31:28.548807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.548459    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:31:28.558836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.555921    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:31:28.610514 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:28.610437    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:31:28.708835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.708796    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:28.712716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:28.709384    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:29.007530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.007475    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:29.010442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.010412    5028 generic.go:356] "Generic (PLEG): container finished" podID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerID="d0be13a468bec3d5782a4e5b943c01485ea4bbadba947a8c82952e8970550970" exitCode=0
Oct 30 22:31:29.010771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.010749    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" event={"ID":"5ec93019-94ab-409f-8d83-a9f449a03ef6","Type":"ContainerDied","Data":"d0be13a468bec3d5782a4e5b943c01485ea4bbadba947a8c82952e8970550970"}
Oct 30 22:31:29.015632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.015597    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" event={"ID":"f6de087b-d027-4356-a2a2-4e55b5a258cf","Type":"ContainerStarted","Data":"5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55"}
Oct 30 22:31:29.555857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.555755    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:29.600764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.600584    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:29.601010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.600980    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 22:31:29.720923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.720871    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:29.721536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.721499    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:29.745907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.745863    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:31:29.902785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.902007    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:29.902785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.902074    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:29.958391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.958355    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:31:29.958669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:29.958649    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"]
Oct 30 22:31:30.057464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.054913    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:30.086730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.080479    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:31:30.112520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.112442    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"]
Oct 30 22:31:30.112803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.112785    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"]
Oct 30 22:31:30.135381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.114141    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy"
Oct 30 22:31:30.191054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.191012    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy"
Oct 30 22:31:30.191742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.191717    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" containerName="istio-proxy"
Oct 30 22:31:30.191982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.136861    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.193462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.184586    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.211816    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.211885    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212025    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212097    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212252    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212329    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212361    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212478    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs\") pod \"5ec93019-94ab-409f-8d83-a9f449a03ef6\" (UID: \"5ec93019-94ab-409f-8d83-a9f449a03ef6\") "
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.212882    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdvv\" (UniqueName: \"kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv\") pod \"payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.226712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.213262    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.250115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.240268    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.250115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.240777    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:30.250115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.241529    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.251496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.251463    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.263792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.262328    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:31:30.263792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.262741    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.408769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.406581    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c3c2cf\""
Oct 30 22:31:30.463765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.444728    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "kube-api-access-wmnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:30.463765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.444922    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:30.463765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.444989    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.548805    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560135    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560365    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdvv\" (UniqueName: \"kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv\") pod \"payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560641    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560660    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560700    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec93019-94ab-409f-8d83-a9f449a03ef6-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560719    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560733    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnb4\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-kube-api-access-wmnb4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.573712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.560749    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.620714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.614267    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.681790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.670301    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:30.732717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.712328    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.732717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.713656    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:30.733279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.732894    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdvv\" (UniqueName: \"kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv\") pod \"payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.809014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.808974    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:30.949237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.876253    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82" event={"ID":"5ec93019-94ab-409f-8d83-a9f449a03ef6","Type":"ContainerDied","Data":"09700dbda5f37677c94f2dbcc084382d5934435342c9fc50e73a551a016aa894"}
Oct 30 22:31:30.949237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.876503    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"]
Oct 30 22:31:30.949237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.886979    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.949237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.897908    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:30.949237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.898933    5028 scope.go:122] "RemoveContainer" containerID="d0be13a468bec3d5782a4e5b943c01485ea4bbadba947a8c82952e8970550970"
Oct 30 22:31:30.970541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.960918    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"
Oct 30 22:31:30.970541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.962553    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy" pod "5ec93019-94ab-409f-8d83-a9f449a03ef6" (UID: "5ec93019-94ab-409f-8d83-a9f449a03ef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.970541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.965149    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c3c2cf-dockercfg-tt2hn\""
Oct 30 22:31:30.989739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.988815    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:30.989739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.988894    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:31.003713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.999775    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxnbp\" (UniqueName: \"kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp\") pod \"payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.003713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:30.999844    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.003713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.000359    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec93019-94ab-409f-8d83-a9f449a03ef6-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:31.011718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.004156    5028 generic.go:356] "Generic (PLEG): container finished" podID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerID="755e196a5498dd8ceaff6e58ea50e72febb305af7e5a9a58cfa300f9311c28e6" exitCode=0
Oct 30 22:31:31.011718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.006009    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" event={"ID":"5277e7ea-ef19-4d91-bdf7-2603d56f7a70","Type":"ContainerDied","Data":"755e196a5498dd8ceaff6e58ea50e72febb305af7e5a9a58cfa300f9311c28e6"}
Oct 30 22:31:31.011718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.007860    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:31.011718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.007908    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:31.159072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.055176    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:31.159072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.056204    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:31:31.159072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.091153    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:31.159072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.103330    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxnbp\" (UniqueName: \"kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp\") pod \"payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.159072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.103399    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.222085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.220966    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:31:31.311153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.311115    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:31.311459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.311423    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 22:31:31.329837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.329788    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:31:31.484771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.478339    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.553713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.508938    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxnbp\" (UniqueName: \"kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp\") pod \"payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.609091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.609042    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:31.699926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.699876    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:31:31.709002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.708967    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:31.826106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.826073    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:31.826713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.826626    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:31.879091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.879057    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:31.882109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.879366    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:31.918520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.918127    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:31.962722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.951070    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:31:31.999836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:31.983246    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095175    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095238    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095351    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095392    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095543    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095580    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095642    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.095712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095705    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.096355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.095975    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs\") pod \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\" (UID: \"5277e7ea-ef19-4d91-bdf7-2603d56f7a70\") "
Oct 30 22:31:32.154711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.153091    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:32.154711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.154314    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:31:32.154711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.154371    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:31:32.154711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.154402    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:31:32.402537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.402329    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:32.403139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.402887    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:32.403308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.403287    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:32.409064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.409030    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.409255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.409223    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.409386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.409372    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.409488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.409473    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.445043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.444814    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:32.491721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.476965    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:32.491721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.478973    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:31:32.491721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.479030    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 22:31:32.491721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.490034    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "kube-api-access-vfs5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:32.561922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.561830    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:32.609534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.609492    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:32.624711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.614519    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.625062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.625017    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.625207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.625191    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfs5r\" (UniqueName: \"kubernetes.io/projected/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-kube-api-access-vfs5r\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.625322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.625310    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.777724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.767884    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo" pod "5277e7ea-ef19-4d91-bdf7-2603d56f7a70" (UID: "5277e7ea-ef19-4d91-bdf7-2603d56f7a70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:32.777724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.768018    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:31:32.777724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.768068    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 22:31:32.834458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.834411    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5277e7ea-ef19-4d91-bdf7-2603d56f7a70-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:32.901768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.898344    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:32.943227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.943184    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:32.943402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.922799    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:31:32.943977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.943950    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:31:32.987707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.987475    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:32.996853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:32.987975    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:31:33.017790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.017307    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:31:33.045680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.041087    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:33.059636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.059608    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"
Oct 30 22:31:33.060742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.060712    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d" event={"ID":"5277e7ea-ef19-4d91-bdf7-2603d56f7a70","Type":"ContainerDied","Data":"bf5537473e90f17e2520a750f6e4532a662f005883e459cfc069926def17fc1e"}
Oct 30 22:31:33.070957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.070921    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:31:33.072060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.072041    5028 scope.go:122] "RemoveContainer" containerID="755e196a5498dd8ceaff6e58ea50e72febb305af7e5a9a58cfa300f9311c28e6"
Oct 30 22:31:33.107935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.107890    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:33.114984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.114550    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:31:33.192451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.190995    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"]
Oct 30 22:31:33.199398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.198703    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-8sf82"]
Oct 30 22:31:33.206954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.206916    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"]
Oct 30 22:31:33.488464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.487048    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec93019-94ab-409f-8d83-a9f449a03ef6" path="/var/lib/kubelet/pods/5ec93019-94ab-409f-8d83-a9f449a03ef6/volumes"
Oct 30 22:31:33.593956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.589244    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:33.593956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.589542    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:31:33.602895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.602863    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"]
Oct 30 22:31:33.603725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.603678    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bzv4d"]
Oct 30 22:31:33.605854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.604767    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:31:33.709036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.709005    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:31:33.709402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.709381    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:31:33.710913 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.710701    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:31:33.812364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.812332    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:33.812534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.812385    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:33.912895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.912808    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:33.966599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:33.966501    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"]
Oct 30 22:31:34.104345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.104305    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96" event={"ID":"096b8c19-2b8c-469b-a00f-8c458fb8fac3","Type":"ContainerStarted","Data":"2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e"}
Oct 30 22:31:34.104627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.104601    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96" event={"ID":"096b8c19-2b8c-469b-a00f-8c458fb8fac3","Type":"ContainerStarted","Data":"460e83207cf9f390ea6f56000998e0098d33bffdbf2a8b5b71eb77480160925d"}
Oct 30 22:31:34.105981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.105958    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:34.110642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.110612    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb" event={"ID":"39d72902-7182-4525-ac29-0000ebfd5d65","Type":"ContainerStarted","Data":"ee27b45e6c998033139819a20adedf2d8c0c7cfea327610ec4317bbb06e5cc19"}
Oct 30 22:31:34.408707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.404811    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96" podStartSLOduration=5.404777144 podStartE2EDuration="5.404777144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:34.243754558 +0000 UTC m=+1825.093267546" watchObservedRunningTime="2026-10-30 22:31:34.404777144 +0000 UTC m=+1825.254290120"
Oct 30 22:31:34.670163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.670088    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"]
Oct 30 22:31:34.670984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.670965    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy"
Oct 30 22:31:34.670984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.670982    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy"
Oct 30 22:31:34.671179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.671100    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" containerName="istio-proxy"
Oct 30 22:31:34.673767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.673749    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.741252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.741195    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6116bd-openshift-default-dockercfg-kmrm9\""
Oct 30 22:31:34.755599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.755518    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"]
Oct 30 22:31:34.784137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.784041    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:34.822717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.819101    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:34.822717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.819165    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853188    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853279    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853410    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853456    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853484    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853523    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853583    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853631    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddzj\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.878739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.853702    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.958819    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.958913    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959064    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959116    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959142    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959181    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959247    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959300    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddzj\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.959358    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.960787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.960330    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.967714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.962168    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.967714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.962425    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.967714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.962660    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.967714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.963921    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.974396 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.974371    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:34.980149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:34.980118    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:35.066645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.066393    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:35.076717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.074840    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddzj\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj\") pod \"e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") " pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:35.108436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.107980    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.158887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.158806    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb" event={"ID":"39d72902-7182-4525-ac29-0000ebfd5d65","Type":"ContainerStarted","Data":"e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4"}
Oct 30 22:31:35.159607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.159240    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:35.271108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.271000    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb" podStartSLOduration=6.270985331 podStartE2EDuration="6.270985331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:35.269848458 +0000 UTC m=+1826.119361421" watchObservedRunningTime="2026-10-30 22:31:35.270985331 +0000 UTC m=+1826.120498303"
Oct 30 22:31:35.286630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.286588    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:35.294097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.294066    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5277e7ea-ef19-4d91-bdf7-2603d56f7a70" path="/var/lib/kubelet/pods/5277e7ea-ef19-4d91-bdf7-2603d56f7a70/volumes"
Oct 30 22:31:35.763130 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:31:35.763079    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b31fa1_9460_47b4_bddc_080508b9c5b7.slice/crio-09374de662e289a7ea2f9a9132d071a079ea3a07915e6cb408f9661a867eb4e5 WatchSource:0}: Error finding container 09374de662e289a7ea2f9a9132d071a079ea3a07915e6cb408f9661a867eb4e5: Status 404 returned error can't find the container with id 09374de662e289a7ea2f9a9132d071a079ea3a07915e6cb408f9661a867eb4e5
Oct 30 22:31:35.763637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.763612    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"]
Oct 30 22:31:35.766908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.766864    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:35.767008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.766965    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:35.767070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.767014    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:35.812787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.812753    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:31:35.812987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:35.812822    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:31:36.169940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.169839    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" event={"ID":"74b31fa1-9460-47b4-bddc-080508b9c5b7","Type":"ContainerStarted","Data":"bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"}
Oct 30 22:31:36.169940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.169905    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" event={"ID":"74b31fa1-9460-47b4-bddc-080508b9c5b7","Type":"ContainerStarted","Data":"09374de662e289a7ea2f9a9132d071a079ea3a07915e6cb408f9661a867eb4e5"}
Oct 30 22:31:36.287129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.287076    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podStartSLOduration=2.2870622640000002 podStartE2EDuration="2.287062264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:36.286085965 +0000 UTC m=+1827.135598939" watchObservedRunningTime="2026-10-30 22:31:36.287062264 +0000 UTC m=+1827.136575237"
Oct 30 22:31:36.287358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.287264    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:36.289129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.289100    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:31:36.289251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.289149    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:31:36.816529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.816495    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:31:36.817031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.817006    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:31:36.817805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:36.817779    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:31:37.295194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:37.295164    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:38.190208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:38.190156    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:38.191207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:38.191184    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:31:39.398737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:39.398649    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:39.687100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:39.686991    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:39.880093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:39.880056    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:40.048150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:40.048111    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.137810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.137778    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.289984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.289946    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.488681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.488645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.685487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.685450    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"]
Oct 30 22:31:44.689651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.689629    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:44.703484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.703455    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75555\" (UniqueName: \"kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555\") pod \"payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:44.703610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.703500    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:44.783790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.781665    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:44.792170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.792141    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6116bd\""
Oct 30 22:31:44.792170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.792165    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6116bd-dockercfg-44tnn\""
Oct 30 22:31:44.799896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.799868    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"]
Oct 30 22:31:44.805483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.805449    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75555\" (UniqueName: \"kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555\") pod \"payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:44.805596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.805510    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:44.806524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.806497    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:44.866379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:44.866344    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75555\" (UniqueName: \"kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555\") pod \"payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:45.006802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:45.005774    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:45.577243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:45.577202    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"]
Oct 30 22:31:46.170053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.170020    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:46.173846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.173817    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:31:46.174097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.174081    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:31:46.302172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.301249    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg" event={"ID":"1b1c3a7b-f659-4491-976b-255f4448f14e","Type":"ContainerStarted","Data":"6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240"}
Oct 30 22:31:46.302172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.301320    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg" event={"ID":"1b1c3a7b-f659-4491-976b-255f4448f14e","Type":"ContainerStarted","Data":"f37ce1def457d3b43ca99aae10bf15104cff206cad0de0d90f219ef77c8d24f5"}
Oct 30 22:31:46.302474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.302452    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:46.368589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.368553    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:46.424871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:46.424757    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg" podStartSLOduration=2.42473759 podStartE2EDuration="2.42473759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:46.421851174 +0000 UTC m=+1837.271364162" watchObservedRunningTime="2026-10-30 22:31:46.42473759 +0000 UTC m=+1837.274250569"
Oct 30 22:31:49.156251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:49.156209    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:50.391180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:50.391142    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:50.850059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:50.850013    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:51.012596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:51.012530    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:51.177076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:51.176972    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:52.147583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:52.147516    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:52.397252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:52.397214    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:54.123807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:54.123776    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:54.387677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:54.387596    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:54.608826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:54.608789    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:56.848758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:56.848720    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:57.022794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:57.022757    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:57.180242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:57.180150    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:58.414484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:58.414454    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:31:58.438216 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:31:58.438179    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:31:58.681108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:31:58.681031    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:00.045842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:00.045795    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:00.260830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:00.260794    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:00.428321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:00.428181    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:01.126846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:01.126811    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:01.385106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:01.385017    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:01.582286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:01.582247    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"]
Oct 30 22:32:01.629781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:01.629744    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61346ee8-rc9gj"]
Oct 30 22:32:02.026611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:02.026575    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.294152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:03.294119    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f484b5-807c-466f-9606-af960b86b1f9" path="/var/lib/kubelet/pods/02f484b5-807c-466f-9606-af960b86b1f9/volumes"
Oct 30 22:32:05.251716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.251649    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:05.507036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.506941    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"]
Oct 30 22:32:05.512094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.512074    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.565007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.564968    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.565258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.565117    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9g6\" (UniqueName: \"kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.565349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.565312    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.574024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.573991    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"]
Oct 30 22:32:05.613028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.612995    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:05.613246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.613039    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:05.613246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.613122    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:32:05.613246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.613155    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:05.667256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.667215    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.667434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.667341    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9g6\" (UniqueName: \"kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.667514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.667498    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.668060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.668028    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.670197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.670173    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.727772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.727738    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9g6\" (UniqueName: \"kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:05.830977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:05.830887    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:06.217533 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:32:06.217501    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450333de_01c3_4fb2_a806_9f9f1ea43fb7.slice/crio-ab059f3cdb0aa7642c84d883d1d1fe19200e6d095064490cf65fb8794c60c869 WatchSource:0}: Error finding container ab059f3cdb0aa7642c84d883d1d1fe19200e6d095064490cf65fb8794c60c869: Status 404 returned error can't find the container with id ab059f3cdb0aa7642c84d883d1d1fe19200e6d095064490cf65fb8794c60c869
Oct 30 22:32:06.217803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:06.217781    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"]
Oct 30 22:32:06.889651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:06.889546    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m" event={"ID":"450333de-01c3-4fb2-a806-9f9f1ea43fb7","Type":"ContainerStarted","Data":"881833bb02281404fdb2a4df039b7917898c29466c6ba9cd7d5fa920e1f59d5c"}
Oct 30 22:32:06.889651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:06.889623    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m" event={"ID":"450333de-01c3-4fb2-a806-9f9f1ea43fb7","Type":"ContainerStarted","Data":"ab059f3cdb0aa7642c84d883d1d1fe19200e6d095064490cf65fb8794c60c869"}
Oct 30 22:32:07.004259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:07.004187    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m" podStartSLOduration=2.004172355 podStartE2EDuration="2.004172355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:07.000236203 +0000 UTC m=+1857.849749201" watchObservedRunningTime="2026-10-30 22:32:07.004172355 +0000 UTC m=+1857.853685327"
Oct 30 22:32:07.900861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:07.900827    5028 generic.go:356] "Generic (PLEG): container finished" podID="450333de-01c3-4fb2-a806-9f9f1ea43fb7" containerID="881833bb02281404fdb2a4df039b7917898c29466c6ba9cd7d5fa920e1f59d5c" exitCode=0
Oct 30 22:32:07.901365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:07.900923    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m" event={"ID":"450333de-01c3-4fb2-a806-9f9f1ea43fb7","Type":"ContainerDied","Data":"881833bb02281404fdb2a4df039b7917898c29466c6ba9cd7d5fa920e1f59d5c"}
Oct 30 22:32:08.872108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:08.872073    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:09.046264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.046160    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:09.151932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.151904    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:09.215737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.215704    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"]
Oct 30 22:32:09.215957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.215934    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy" containerID="cri-o://94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601" gracePeriod=30
Oct 30 22:32:09.224077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.224049    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca\") pod \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") "
Oct 30 22:32:09.224363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.224341    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca\") pod \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") "
Oct 30 22:32:09.224790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.224677    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:09.224790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.224730    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca" pod "450333de-01c3-4fb2-a806-9f9f1ea43fb7" (UID: "450333de-01c3-4fb2-a806-9f9f1ea43fb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:09.224916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.224851    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6\" (UniqueName: \"kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6\") pod \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\" (UID: \"450333de-01c3-4fb2-a806-9f9f1ea43fb7\") "
Oct 30 22:32:09.225649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.225625    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.226552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.226527    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca" pod "450333de-01c3-4fb2-a806-9f9f1ea43fb7" (UID: "450333de-01c3-4fb2-a806-9f9f1ea43fb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:09.228001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.227977    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6" pod "450333de-01c3-4fb2-a806-9f9f1ea43fb7" (UID: "450333de-01c3-4fb2-a806-9f9f1ea43fb7"). InnerVolumeSpecName "kube-api-access-kl9g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:09.349771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.341531    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450333de-01c3-4fb2-a806-9f9f1ea43fb7-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.349771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.341568    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9g6\" (UniqueName: \"kubernetes.io/projected/450333de-01c3-4fb2-a806-9f9f1ea43fb7-kube-api-access-kl9g6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.947057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.946975    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"
Oct 30 22:32:09.947057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.946993    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m" event={"ID":"450333de-01c3-4fb2-a806-9f9f1ea43fb7","Type":"ContainerDied","Data":"ab059f3cdb0aa7642c84d883d1d1fe19200e6d095064490cf65fb8794c60c869"}
Oct 30 22:32:09.947057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:09.947057    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab059f3cdb0aa7642c84d883d1d1fe19200e6d095064490cf65fb8794c60c869"
Oct 30 22:32:10.179930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:10.179894    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:10.365710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:10.365491    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:10.535438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:10.535406    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:11.744586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:11.744552    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:11.906433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:11.906395    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:14.527287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.527259    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:32:14.578145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578112    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578183    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578238    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578268    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578306    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578376    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578416    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578563    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578563    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:14.578787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578616    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs\") pod \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\" (UID: \"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7\") "
Oct 30 22:32:14.578787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578722    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:14.578893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578793    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:14.578946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.578901    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:14.579067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.579047    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:14.579360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.579340    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.579360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.579360    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.579530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.579369    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.579530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.579378    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.579530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.579387    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.580734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.580676    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:14.580852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.580835    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "kube-api-access-cbdtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:14.580974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.580955    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:14.581034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.581015    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo" pod "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" (UID: "26e6dd8e-8361-4624-9ff2-7cc06b2f11e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:14.681094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.681055    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.681094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.681090    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdtb\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-kube-api-access-cbdtb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.681094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.681100    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.681350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.681109    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:14.850134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:14.850055    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:15.001829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.001788    5028 generic.go:356] "Generic (PLEG): container finished" podID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerID="94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601" exitCode=0
Oct 30 22:32:15.002035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.001845    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"
Oct 30 22:32:15.002035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.001870    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" event={"ID":"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7","Type":"ContainerDied","Data":"94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601"}
Oct 30 22:32:15.002035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.001923    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659" event={"ID":"26e6dd8e-8361-4624-9ff2-7cc06b2f11e7","Type":"ContainerDied","Data":"cde607ae9e3f6025d92d73f664494b00981af46b2307578b8e63625c62991d95"}
Oct 30 22:32:15.002035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.001948    5028 scope.go:122] "RemoveContainer" containerID="94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601"
Oct 30 22:32:15.018515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.018489    5028 scope.go:122] "RemoveContainer" containerID="e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"
Oct 30 22:32:15.031042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.031019    5028 scope.go:122] "RemoveContainer" containerID="94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601"
Oct 30 22:32:15.031374 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:32:15.031345    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601\": container with ID starting with 94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601 not found: ID does not exist" containerID="94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601"
Oct 30 22:32:15.031500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.031381    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601"} err="failed to get container status \"94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601\": rpc error: code = NotFound desc = could not find container \"94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601\": container with ID starting with 94be8564400d8aea70e9f2c75f0e584ba11b3c718c5a0435b019d53b86fb4601 not found: ID does not exist"
Oct 30 22:32:15.031500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.031398    5028 scope.go:122] "RemoveContainer" containerID="e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"
Oct 30 22:32:15.031700 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:32:15.031657    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68\": container with ID starting with e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68 not found: ID does not exist" containerID="e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"
Oct 30 22:32:15.031780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.031736    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68"} err="failed to get container status \"e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68\": rpc error: code = NotFound desc = could not find container \"e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68\": container with ID starting with e46dfd3be2eaab7947f225e9776d89d49dccc0a7ef89758c6c1e2d7a8c9a9a68 not found: ID does not exist"
Oct 30 22:32:15.164050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.163958    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"]
Oct 30 22:32:15.168470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.168447    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-61346ee8-openshift-default-c79649cc-r8659"]
Oct 30 22:32:15.295274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:15.295242    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" path="/var/lib/kubelet/pods/26e6dd8e-8361-4624-9ff2-7cc06b2f11e7/volumes"
Oct 30 22:32:16.953275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:16.953242    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"]
Oct 30 22:32:16.998565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:16.998534    5028 scope.go:122] "RemoveContainer" containerID="8934d9f8f08f86adc47b822a944779fddc15e2885ceb68f9ecbc14c019d13233"
Oct 30 22:32:17.009733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:17.009710    5028 scope.go:122] "RemoveContainer" containerID="1c3f3a1cb7b58591ebdc81d8d68ebd0810bc9e52601886ede0f682036f8de2a1"
Oct 30 22:32:17.050896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:17.050870    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"]
Oct 30 22:32:17.328386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:17.328342    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.489608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:17.489574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:18.977668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.977636    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"]
Oct 30 22:32:18.978915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.978892    5028 cpu_manager.go:414] "containerMap: removing container" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy"
Oct 30 22:32:18.978971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.978918    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy"
Oct 30 22:32:18.978971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.978962    5028 cpu_manager.go:414] "containerMap: removing container" podUID="450333de-01c3-4fb2-a806-9f9f1ea43fb7" containerName="revoke-keys"
Oct 30 22:32:18.979037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.978978    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450333de-01c3-4fb2-a806-9f9f1ea43fb7" containerName="revoke-keys"
Oct 30 22:32:18.979071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.979044    5028 cpu_manager.go:414] "containerMap: removing container" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy"
Oct 30 22:32:18.979071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.979053    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy"
Oct 30 22:32:18.979247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.979236    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="450333de-01c3-4fb2-a806-9f9f1ea43fb7" containerName="revoke-keys"
Oct 30 22:32:18.979283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.979256    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy"
Oct 30 22:32:18.979283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.979268    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e6dd8e-8361-4624-9ff2-7cc06b2f11e7" containerName="istio-proxy"
Oct 30 22:32:18.985182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:18.985161    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.043747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.043709    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"]
Oct 30 22:32:19.080595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.080560    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlc5\" (UniqueName: \"kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.080978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.080952    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.081118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.081098    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.082291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.082261    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:19.082456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.082294    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:19.082456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.082317    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:32:19.082456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.082423    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:19.182540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.182507    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlc5\" (UniqueName: \"kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.182802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.182700    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.182802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.182763    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.183556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.183531    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.185296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.185258    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.242396 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.242323    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlc5\" (UniqueName: \"kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5\") pod \"maas-api-revoke-sub-63e0aaca-dxkcw\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.300899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.300864    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:19.657022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.656920    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:19.683628 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:32:19.683600    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4ac25f_59b3_4334_bef5_ed89ae2de00e.slice/crio-e9f840b2315ba317d2eff0a7366d270a8800a0bd19c990194a1ac9305ad85f08 WatchSource:0}: Error finding container e9f840b2315ba317d2eff0a7366d270a8800a0bd19c990194a1ac9305ad85f08: Status 404 returned error can't find the container with id e9f840b2315ba317d2eff0a7366d270a8800a0bd19c990194a1ac9305ad85f08
Oct 30 22:32:19.684780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:19.684753    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"]
Oct 30 22:32:20.058175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:20.058140    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw" event={"ID":"ad4ac25f-59b3-4334-bef5-ed89ae2de00e","Type":"ContainerStarted","Data":"eaf5f583b22920bdd6bfd8e316928c3956dae7d27cb3e26bebe8bbd0573c28bd"}
Oct 30 22:32:20.058728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:20.058206    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw" event={"ID":"ad4ac25f-59b3-4334-bef5-ed89ae2de00e","Type":"ContainerStarted","Data":"e9f840b2315ba317d2eff0a7366d270a8800a0bd19c990194a1ac9305ad85f08"}
Oct 30 22:32:20.170251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:20.170199    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw" podStartSLOduration=2.170185173 podStartE2EDuration="2.170185173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:20.169169177 +0000 UTC m=+1871.018682151" watchObservedRunningTime="2026-10-30 22:32:20.170185173 +0000 UTC m=+1871.019698146"
Oct 30 22:32:20.488759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:20.488725    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:20.644816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:20.644775    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:21.069622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:21.069587    5028 generic.go:356] "Generic (PLEG): container finished" podID="ad4ac25f-59b3-4334-bef5-ed89ae2de00e" containerID="eaf5f583b22920bdd6bfd8e316928c3956dae7d27cb3e26bebe8bbd0573c28bd" exitCode=0
Oct 30 22:32:21.070038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:21.069670    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw" event={"ID":"ad4ac25f-59b3-4334-bef5-ed89ae2de00e","Type":"ContainerDied","Data":"eaf5f583b22920bdd6bfd8e316928c3956dae7d27cb3e26bebe8bbd0573c28bd"}
Oct 30 22:32:21.953543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:21.953504    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerName="payload-processing" containerID="cri-o://ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42" gracePeriod=25
Oct 30 22:32:22.051777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.051731    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" containerName="payload-pre-processing" containerID="cri-o://248e78baca2da53758f111de9cd20730afb7ed8b862909bff9bc347c5af5f0b5" gracePeriod=25
Oct 30 22:32:22.285058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.285033    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:22.432649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.432615    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5\" (UniqueName: \"kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5\") pod \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") "
Oct 30 22:32:22.432862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.432679    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca\") pod \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") "
Oct 30 22:32:22.432930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.432890    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca\") pod \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\" (UID: \"ad4ac25f-59b3-4334-bef5-ed89ae2de00e\") "
Oct 30 22:32:22.433509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.433477    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca" pod "ad4ac25f-59b3-4334-bef5-ed89ae2de00e" (UID: "ad4ac25f-59b3-4334-bef5-ed89ae2de00e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:22.435566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.435536    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5" pod "ad4ac25f-59b3-4334-bef5-ed89ae2de00e" (UID: "ad4ac25f-59b3-4334-bef5-ed89ae2de00e"). InnerVolumeSpecName "kube-api-access-xrlc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:22.436052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.436021    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca" pod "ad4ac25f-59b3-4334-bef5-ed89ae2de00e" (UID: "ad4ac25f-59b3-4334-bef5-ed89ae2de00e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:22.535329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.535248    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlc5\" (UniqueName: \"kubernetes.io/projected/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-kube-api-access-xrlc5\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:22.535329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.535281    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:22.535329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:22.535292    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4ac25f-59b3-4334-bef5-ed89ae2de00e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:23.088574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:23.088543    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"
Oct 30 22:32:23.088814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:23.088577    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw" event={"ID":"ad4ac25f-59b3-4334-bef5-ed89ae2de00e","Type":"ContainerDied","Data":"e9f840b2315ba317d2eff0a7366d270a8800a0bd19c990194a1ac9305ad85f08"}
Oct 30 22:32:23.088814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:23.088625    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9f840b2315ba317d2eff0a7366d270a8800a0bd19c990194a1ac9305ad85f08"
Oct 30 22:32:24.258415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:24.258373    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.438970 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:32:28.438920    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:32:28.639871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:28.639825    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.790729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:28.790675    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.948877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:28.948837    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:29.253442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:29.253391    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:29.253624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:29.253448    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:29.745822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:29.745790    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:33.659765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:33.659668    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:36.467214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:36.467174    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.240376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:38.240325    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.253725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:39.253663    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:39.254125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:39.253682    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:42.285810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.283165    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:32:42.379401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.375489    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4\" (UniqueName: \"kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4\") pod \"f3b49da8-b3c0-46fa-8168-904081b0ad22\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") "
Oct 30 22:32:42.379401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.375754    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume\") pod \"f3b49da8-b3c0-46fa-8168-904081b0ad22\" (UID: \"f3b49da8-b3c0-46fa-8168-904081b0ad22\") "
Oct 30 22:32:42.379401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.377444    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume" pod "f3b49da8-b3c0-46fa-8168-904081b0ad22" (UID: "f3b49da8-b3c0-46fa-8168-904081b0ad22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:42.379401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.378591    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4" pod "f3b49da8-b3c0-46fa-8168-904081b0ad22" (UID: "f3b49da8-b3c0-46fa-8168-904081b0ad22"). InnerVolumeSpecName "kube-api-access-g5zc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:42.381385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.381359    5028 generic.go:356] "Generic (PLEG): container finished" podID="c808208d-bc25-45d2-8c33-33c1d4823300" containerID="248e78baca2da53758f111de9cd20730afb7ed8b862909bff9bc347c5af5f0b5" exitCode=0
Oct 30 22:32:42.381498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.381395    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" event={"ID":"c808208d-bc25-45d2-8c33-33c1d4823300","Type":"ContainerDied","Data":"248e78baca2da53758f111de9cd20730afb7ed8b862909bff9bc347c5af5f0b5"}
Oct 30 22:32:42.381498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.381462    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn" event={"ID":"c808208d-bc25-45d2-8c33-33c1d4823300","Type":"ContainerDied","Data":"e332562d9ebb9893855735adab0e33e7672705e2e2493fcc14535b53033adb87"}
Oct 30 22:32:42.381498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.381488    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e332562d9ebb9893855735adab0e33e7672705e2e2493fcc14535b53033adb87"
Oct 30 22:32:42.383351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.383329    5028 generic.go:356] "Generic (PLEG): container finished" podID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerID="ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42" exitCode=0
Oct 30 22:32:42.383451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.383387    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"
Oct 30 22:32:42.383451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.383419    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" event={"ID":"f3b49da8-b3c0-46fa-8168-904081b0ad22","Type":"ContainerDied","Data":"ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42"}
Oct 30 22:32:42.383590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.383479    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b" event={"ID":"f3b49da8-b3c0-46fa-8168-904081b0ad22","Type":"ContainerDied","Data":"dba7c5ee17c02fb37a24ec6990e7689c5108fcd034988482bbbba2a46a0a270d"}
Oct 30 22:32:42.383590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.383515    5028 scope.go:122] "RemoveContainer" containerID="ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42"
Oct 30 22:32:42.395447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.395432    5028 scope.go:122] "RemoveContainer" containerID="ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42"
Oct 30 22:32:42.395788 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:32:42.395767    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42\": container with ID starting with ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42 not found: ID does not exist" containerID="ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42"
Oct 30 22:32:42.395862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.395798    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42"} err="failed to get container status \"ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42\": rpc error: code = NotFound desc = could not find container \"ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42\": container with ID starting with ca31d0a19cc8dd2124e2fe0766daa4b62b065ae4085ed6fecbe5c02cbbf61b42 not found: ID does not exist"
Oct 30 22:32:42.397344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.397328    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:32:42.477658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.477624    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume\") pod \"c808208d-bc25-45d2-8c33-33c1d4823300\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") "
Oct 30 22:32:42.477854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.477744    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4\" (UniqueName: \"kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4\") pod \"c808208d-bc25-45d2-8c33-33c1d4823300\" (UID: \"c808208d-bc25-45d2-8c33-33c1d4823300\") "
Oct 30 22:32:42.478040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.478014    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume" pod "c808208d-bc25-45d2-8c33-33c1d4823300" (UID: "c808208d-bc25-45d2-8c33-33c1d4823300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:42.478339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.478324    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808208d-bc25-45d2-8c33-33c1d4823300-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.478415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.478341    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5zc4\" (UniqueName: \"kubernetes.io/projected/f3b49da8-b3c0-46fa-8168-904081b0ad22-kube-api-access-g5zc4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.478415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.478350    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b49da8-b3c0-46fa-8168-904081b0ad22-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.479907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.479883    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4" pod "c808208d-bc25-45d2-8c33-33c1d4823300" (UID: "c808208d-bc25-45d2-8c33-33c1d4823300"). InnerVolumeSpecName "kube-api-access-b9fv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:42.512340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.512265    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"]
Oct 30 22:32:42.517033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.517007    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-l9d5b"]
Oct 30 22:32:42.579502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.579469    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fv4\" (UniqueName: \"kubernetes.io/projected/c808208d-bc25-45d2-8c33-33c1d4823300-kube-api-access-b9fv4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.758166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:42.758121    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.293702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:43.293617    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" path="/var/lib/kubelet/pods/f3b49da8-b3c0-46fa-8168-904081b0ad22/volumes"
Oct 30 22:32:43.392695 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:43.392666    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"
Oct 30 22:32:43.562617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:43.562524    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"]
Oct 30 22:32:43.609575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:43.609544    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-g2xjn"]
Oct 30 22:32:45.293859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:45.293825    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" path="/var/lib/kubelet/pods/c808208d-bc25-45d2-8c33-33c1d4823300/volumes"
Oct 30 22:32:47.032949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.032912    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"]
Oct 30 22:32:47.033889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.033865    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" containerName="payload-pre-processing"
Oct 30 22:32:47.033889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.033887    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" containerName="payload-pre-processing"
Oct 30 22:32:47.034128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.033919    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerName="payload-processing"
Oct 30 22:32:47.034128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.033927    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerName="payload-processing"
Oct 30 22:32:47.034128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.033950    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ad4ac25f-59b3-4334-bef5-ed89ae2de00e" containerName="revoke-keys"
Oct 30 22:32:47.034128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.033958    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4ac25f-59b3-4334-bef5-ed89ae2de00e" containerName="revoke-keys"
Oct 30 22:32:47.034346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.034187    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c808208d-bc25-45d2-8c33-33c1d4823300" containerName="payload-pre-processing"
Oct 30 22:32:47.034346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.034207    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b49da8-b3c0-46fa-8168-904081b0ad22" containerName="payload-processing"
Oct 30 22:32:47.034346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.034223    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4ac25f-59b3-4334-bef5-ed89ae2de00e" containerName="revoke-keys"
Oct 30 22:32:47.039943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.039916    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.151187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.151146    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"]
Oct 30 22:32:47.152776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.152738    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.153312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.153285    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.153590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.153561    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8v8p\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.153711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.153652    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.154120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.153716    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.154120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.153816    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.154120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.153846    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.154120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.154003    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.154120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.154042    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.234956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.234911    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.263532    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.264990    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.265159    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.265470    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.275462    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8v8p\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.275787    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.275869    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.276023    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.282039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.276052    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.283185    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.283774    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.287230    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.287384    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.287501    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.287584    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.291510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.290641    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.337724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.335497    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.337724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.336078    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8v8p\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4xwjx\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.358543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.358510    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:47.801759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.798158    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"]
Oct 30 22:32:47.801759 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:32:47.798894    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb653cb_e45d_4edf_80f9_0514e2032093.slice/crio-f5b9e05e9c621aeee02d69d3e743dafe8b282bcfd68adf8f580c0eeacea8c296 WatchSource:0}: Error finding container f5b9e05e9c621aeee02d69d3e743dafe8b282bcfd68adf8f580c0eeacea8c296: Status 404 returned error can't find the container with id f5b9e05e9c621aeee02d69d3e743dafe8b282bcfd68adf8f580c0eeacea8c296
Oct 30 22:32:47.801759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.801609    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:47.801759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.801706    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:47.801759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:47.801751    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:48.458520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:48.458473    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" event={"ID":"9bb653cb-e45d-4edf-80f9-0514e2032093","Type":"ContainerStarted","Data":"dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b"}
Oct 30 22:32:48.459056 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:48.458581    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" event={"ID":"9bb653cb-e45d-4edf-80f9-0514e2032093","Type":"ContainerStarted","Data":"f5b9e05e9c621aeee02d69d3e743dafe8b282bcfd68adf8f580c0eeacea8c296"}
Oct 30 22:32:48.578541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:48.578486    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podStartSLOduration=2.578473812 podStartE2EDuration="2.578473812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:48.578247342 +0000 UTC m=+1899.427760316" watchObservedRunningTime="2026-10-30 22:32:48.578473812 +0000 UTC m=+1899.427986784"
Oct 30 22:32:49.359316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:49.359282    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:49.360897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:49.360871    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:49.361032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:49.360919    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:50.359654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:50.359617    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:50.360209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:50.359679    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:51.359463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:51.359428    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:51.359637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:51.359493    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:51.751149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:51.751112    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.908948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:51.908914    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:52.063617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:52.063533    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:52.359182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:52.359098    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:52.359182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:52.359167    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:53.359462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:53.359422    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:53.360064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:53.359483    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:54.359740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:54.359662    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:54.360179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:54.359761    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:55.359399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:55.359361    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4xwjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body=
Oct 30 22:32:55.359589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:55.359425    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused"
Oct 30 22:32:55.902583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:55.902545    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"]
Oct 30 22:32:55.907268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:55.907244    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:55.968214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:55.968177    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"]
Oct 30 22:32:56.007080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.007042    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:32:56.007282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.007223    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:32:56.007282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.007273    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:32:56.007453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.007271    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:32:56.047412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.047373    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.047621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.047589    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swrc\" (UniqueName: \"kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.047728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.047677    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.149514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.149467    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.149752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.149657    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swrc\" (UniqueName: \"kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.149752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.149738    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.150278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.150252    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.152415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.152389    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.209054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.209017    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swrc\" (UniqueName: \"kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc\") pod \"maas-api-revoke-sub-4234c5fa-cbqh8\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.228749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.228718    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:56.614768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.614271    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"]
Oct 30 22:32:56.614768 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:32:56.614515    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda373be5d_437b_4710_8ddc_9153354d1089.slice/crio-130ccd8c1ef0b1baa62bd53c5a0cc0b767a24688f85c01b72aa000df996159f5 WatchSource:0}: Error finding container 130ccd8c1ef0b1baa62bd53c5a0cc0b767a24688f85c01b72aa000df996159f5: Status 404 returned error can't find the container with id 130ccd8c1ef0b1baa62bd53c5a0cc0b767a24688f85c01b72aa000df996159f5
Oct 30 22:32:56.637609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.637570    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:56.776921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.776890    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:56.777408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.777396    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:56.778215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:56.778196    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:32:57.562184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:57.562140    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8" event={"ID":"a373be5d-437b-4710-8ddc-9153354d1089","Type":"ContainerStarted","Data":"fbc6bbebc09c2effc64e0a65b0d2637e11bbcc3d14cc2d06883cfc2821b8618b"}
Oct 30 22:32:57.562676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:57.562205    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8" event={"ID":"a373be5d-437b-4710-8ddc-9153354d1089","Type":"ContainerStarted","Data":"130ccd8c1ef0b1baa62bd53c5a0cc0b767a24688f85c01b72aa000df996159f5"}
Oct 30 22:32:57.677635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:57.677582    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8" podStartSLOduration=2.677568209 podStartE2EDuration="2.677568209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:57.672914127 +0000 UTC m=+1908.522427130" watchObservedRunningTime="2026-10-30 22:32:57.677568209 +0000 UTC m=+1908.527081181"
Oct 30 22:32:58.437420 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:32:58.437386    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:32:58.573590 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:58.573553    5028 generic.go:356] "Generic (PLEG): container finished" podID="a373be5d-437b-4710-8ddc-9153354d1089" containerID="fbc6bbebc09c2effc64e0a65b0d2637e11bbcc3d14cc2d06883cfc2821b8618b" exitCode=0
Oct 30 22:32:58.574027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:58.573645    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8" event={"ID":"a373be5d-437b-4710-8ddc-9153354d1089","Type":"ContainerDied","Data":"fbc6bbebc09c2effc64e0a65b0d2637e11bbcc3d14cc2d06883cfc2821b8618b"}
Oct 30 22:32:59.771379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.771349    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:32:59.911191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.911105    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc\" (UniqueName: \"kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc\") pod \"a373be5d-437b-4710-8ddc-9153354d1089\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") "
Oct 30 22:32:59.911191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.911179    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca\") pod \"a373be5d-437b-4710-8ddc-9153354d1089\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") "
Oct 30 22:32:59.911572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.911549    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca\") pod \"a373be5d-437b-4710-8ddc-9153354d1089\" (UID: \"a373be5d-437b-4710-8ddc-9153354d1089\") "
Oct 30 22:32:59.912615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.912584    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca" pod "a373be5d-437b-4710-8ddc-9153354d1089" (UID: "a373be5d-437b-4710-8ddc-9153354d1089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:59.914081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.914046    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca" pod "a373be5d-437b-4710-8ddc-9153354d1089" (UID: "a373be5d-437b-4710-8ddc-9153354d1089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:59.914472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:32:59.914436    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc" pod "a373be5d-437b-4710-8ddc-9153354d1089" (UID: "a373be5d-437b-4710-8ddc-9153354d1089"). InnerVolumeSpecName "kube-api-access-2swrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:00.013760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:00.013727    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.013760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:00.013757    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swrc\" (UniqueName: \"kubernetes.io/projected/a373be5d-437b-4710-8ddc-9153354d1089-kube-api-access-2swrc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.013760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:00.013768    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a373be5d-437b-4710-8ddc-9153354d1089-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.595668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:00.595637    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"
Oct 30 22:33:00.595874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:00.595677    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8" event={"ID":"a373be5d-437b-4710-8ddc-9153354d1089","Type":"ContainerDied","Data":"130ccd8c1ef0b1baa62bd53c5a0cc0b767a24688f85c01b72aa000df996159f5"}
Oct 30 22:33:00.595874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:00.595748    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="130ccd8c1ef0b1baa62bd53c5a0cc0b767a24688f85c01b72aa000df996159f5"
Oct 30 22:33:01.060197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:01.060157    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:01.211053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:01.211014    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:01.388285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:01.388201    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:05.756497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:05.756461    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:10.274388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:10.274352    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:10.522541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:10.522505    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:15.036651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:15.036611    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:15.197517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:15.197486    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:15.439002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:15.438915    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:17.101849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:17.101811    5028 scope.go:122] "RemoveContainer" containerID="248e78baca2da53758f111de9cd20730afb7ed8b862909bff9bc347c5af5f0b5"
Oct 30 22:33:19.736769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:19.735810    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:19.878714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:19.878603    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.054787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.054665    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.073712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.064770    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.078921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.078881    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Oct 30 22:33:20.122675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.122637    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:33:20.122978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.122949    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.123173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.123150    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.220236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.219787    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:33:20.328522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.328378    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:20.407154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.407089    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:33:20.741842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.741784    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.744650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.742977    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:33:20.897713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.892926    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:20.897713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:20.894868    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:33:21.130798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.122546    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:21.130798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.122829    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:33:21.134739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.132770    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:21.134739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.133885    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:33:21.134739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.133952    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:33:21.419547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.417730    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:33:21.762713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.754182    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:33:21.762713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.756037    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:33:21.762713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.756097    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:33:21.823820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.823782    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:33:21.885032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.882052    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:33:21.961198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.957979    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:33:21.969391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:21.965397    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:33:24.771850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:24.771803    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:25.116193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:25.116069    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:25.277203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:25.277163    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:27.180198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:27.180161    5028 generic.go:356] "Generic (PLEG): container finished" podID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerID="c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee" exitCode=0
Oct 30 22:33:27.180651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:27.180237    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" event={"ID":"1a3f5d9c-da8a-48b1-a048-17aa76541c4d","Type":"ContainerDied","Data":"c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"}
Oct 30 22:33:27.180853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:27.180835    5028 scope.go:122] "RemoveContainer" containerID="c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"
Oct 30 22:33:27.182841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:27.182809    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:33:27.182938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:27.182896    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:33:27.182993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:27.182977    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:33:28.192872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:28.192837    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" event={"ID":"1a3f5d9c-da8a-48b1-a048-17aa76541c4d","Type":"ContainerStarted","Data":"b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe"}
Oct 30 22:33:28.470569 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:33:28.470537    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:33:29.064591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:29.064558    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:33:29.064803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:29.064757    5028 patch_prober.go:29] interesting pod/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:33:29.064869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:29.064802    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:33:29.207947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:29.207916    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:29.367812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:29.367712    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:29.552347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:29.552307    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:30.064840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:30.064798    5028 patch_prober.go:29] interesting pod/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:33:30.065101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:30.064860    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:33:31.064733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:31.064645    5028 patch_prober.go:29] interesting pod/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:33:31.065265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:31.065239    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:33:32.071720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:32.071666    5028 patch_prober.go:29] interesting pod/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:33:32.072362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:32.072329    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:33:33.064889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:33.064851    5028 patch_prober.go:29] interesting pod/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 22:33:33.065149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:33.065122    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 22:33:33.950547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:33.950513    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:34.070848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:34.070816    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:33:34.071308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:34.071286    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:33:34.072131 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:34.072111    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:33:38.433249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:38.433211    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:42.937536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:42.937486    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:47.348126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:47.348081    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:47.514495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:47.514289    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:50.333758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:50.333703    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:50.530561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:50.530494    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:51.953401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:51.953362    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:56.585942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:33:56.585906    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:58.442049 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:33:58.442003    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:34:00.531078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.531038    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"]
Oct 30 22:34:00.531922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.531903    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a373be5d-437b-4710-8ddc-9153354d1089" containerName="revoke-keys"
Oct 30 22:34:00.532002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.531922    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a373be5d-437b-4710-8ddc-9153354d1089" containerName="revoke-keys"
Oct 30 22:34:00.532086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.532076    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a373be5d-437b-4710-8ddc-9153354d1089" containerName="revoke-keys"
Oct 30 22:34:00.536502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.536475    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.593154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.593118    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:00.633660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.633625    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:34:00.633949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.633643    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:00.634237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.633725    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:00.634360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.634211    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:00.640099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.640074    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"]
Oct 30 22:34:00.703161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.703123    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqpp\" (UniqueName: \"kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.703401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.703379    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.703466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.703421    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.773731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.773636    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:00.805218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.805123    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqpp\" (UniqueName: \"kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.805409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.805320    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.805409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.805350    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.806106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.806082    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.807797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.807776    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:00.864160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:00.864121    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqpp\" (UniqueName: \"kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp\") pod \"maas-api-revoke-sub-034ca8d2-4hq9p\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:01.064566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:01.064428    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:01.154165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:01.154135    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:01.542187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:01.542156    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"]
Oct 30 22:34:01.547412 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:34:01.547380    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34db0655_ba58_4f41_ace7_73e550610cb2.slice/crio-fb7a3521cccfe13bd4f00e6fbbbc46ebae588238fb85a0acfaf80d1e6e9db554 WatchSource:0}: Error finding container fb7a3521cccfe13bd4f00e6fbbbc46ebae588238fb85a0acfaf80d1e6e9db554: Status 404 returned error can't find the container with id fb7a3521cccfe13bd4f00e6fbbbc46ebae588238fb85a0acfaf80d1e6e9db554
Oct 30 22:34:01.602449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:01.602418    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p" event={"ID":"34db0655-ba58-4f41-ace7-73e550610cb2","Type":"ContainerStarted","Data":"fb7a3521cccfe13bd4f00e6fbbbc46ebae588238fb85a0acfaf80d1e6e9db554"}
Oct 30 22:34:02.614848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:02.614806    5028 generic.go:356] "Generic (PLEG): container finished" podID="34db0655-ba58-4f41-ace7-73e550610cb2" containerID="81c443ece20f63ca06ebe0e56d4f541f93c9629f3fd213b1bb8259b1d69eb287" exitCode=0
Oct 30 22:34:02.615299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:02.614848    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p" event={"ID":"34db0655-ba58-4f41-ace7-73e550610cb2","Type":"ContainerDied","Data":"81c443ece20f63ca06ebe0e56d4f541f93c9629f3fd213b1bb8259b1d69eb287"}
Oct 30 22:34:03.902559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.902529    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:03.972246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.972211    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp\" (UniqueName: \"kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp\") pod \"34db0655-ba58-4f41-ace7-73e550610cb2\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") "
Oct 30 22:34:03.972462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.972267    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca\") pod \"34db0655-ba58-4f41-ace7-73e550610cb2\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") "
Oct 30 22:34:03.978889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.972623    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca\") pod \"34db0655-ba58-4f41-ace7-73e550610cb2\" (UID: \"34db0655-ba58-4f41-ace7-73e550610cb2\") "
Oct 30 22:34:03.978889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.972716    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca" pod "34db0655-ba58-4f41-ace7-73e550610cb2" (UID: "34db0655-ba58-4f41-ace7-73e550610cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:03.978889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.973242    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:03.978889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.975045    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca" pod "34db0655-ba58-4f41-ace7-73e550610cb2" (UID: "34db0655-ba58-4f41-ace7-73e550610cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:03.979312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:03.979286    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp" pod "34db0655-ba58-4f41-ace7-73e550610cb2" (UID: "34db0655-ba58-4f41-ace7-73e550610cb2"). InnerVolumeSpecName "kube-api-access-grqpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:04.079325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:04.079257    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqpp\" (UniqueName: \"kubernetes.io/projected/34db0655-ba58-4f41-ace7-73e550610cb2-kube-api-access-grqpp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.079325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:04.079298    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34db0655-ba58-4f41-ace7-73e550610cb2-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.733730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:04.723318    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p" event={"ID":"34db0655-ba58-4f41-ace7-73e550610cb2","Type":"ContainerDied","Data":"fb7a3521cccfe13bd4f00e6fbbbc46ebae588238fb85a0acfaf80d1e6e9db554"}
Oct 30 22:34:04.733730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:04.723378    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb7a3521cccfe13bd4f00e6fbbbc46ebae588238fb85a0acfaf80d1e6e9db554"
Oct 30 22:34:04.733730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:04.723450    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"
Oct 30 22:34:05.976959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:05.976921    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:08.201208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.201175    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"]
Oct 30 22:34:08.202070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.202050    5028 cpu_manager.go:414] "containerMap: removing container" podUID="34db0655-ba58-4f41-ace7-73e550610cb2" containerName="revoke-keys"
Oct 30 22:34:08.202128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.202071    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34db0655-ba58-4f41-ace7-73e550610cb2" containerName="revoke-keys"
Oct 30 22:34:08.202219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.202209    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="34db0655-ba58-4f41-ace7-73e550610cb2" containerName="revoke-keys"
Oct 30 22:34:08.206098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.206078    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.264864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.264828    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"]
Oct 30 22:34:08.286318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.286284    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gt6t\" (UniqueName: \"kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.286523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.286357    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.286523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.286473    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.306816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.306784    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:08.306816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.306786    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:08.306816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.306816    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:34:08.307169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.306896    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:08.388166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.388124    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gt6t\" (UniqueName: \"kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.388350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.388218    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.388350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.388321    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.388927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.388902    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.390751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.390729    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.447436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.447405    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gt6t\" (UniqueName: \"kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t\") pod \"maas-api-revoke-keys-e2e-api-a-721c75-9fmtg\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.540225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.538952    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:08.952756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:08.950899    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"]
Oct 30 22:34:09.784932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:09.784895    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg" event={"ID":"b254728e-4059-4043-8c56-fb40628ff45a","Type":"ContainerStarted","Data":"55762b9b2b0acca1d13951820d5522b2041bb590a24035823c1ff5b3fada9faf"}
Oct 30 22:34:09.785328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:09.784981    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg" event={"ID":"b254728e-4059-4043-8c56-fb40628ff45a","Type":"ContainerStarted","Data":"5cbe4bd40a60132b0109a472b3a024c384ff5d1a6ebad2ca881c032b561082e5"}
Oct 30 22:34:09.910957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:09.910903    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg" podStartSLOduration=1.910888233 podStartE2EDuration="1.910888233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:09.906550386 +0000 UTC m=+1980.756063359" watchObservedRunningTime="2026-10-30 22:34:09.910888233 +0000 UTC m=+1980.760401205"
Oct 30 22:34:10.648977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:10.648937    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:10.797536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:10.797497    5028 generic.go:356] "Generic (PLEG): container finished" podID="b254728e-4059-4043-8c56-fb40628ff45a" containerID="55762b9b2b0acca1d13951820d5522b2041bb590a24035823c1ff5b3fada9faf" exitCode=0
Oct 30 22:34:10.798003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:10.797589    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg" event={"ID":"b254728e-4059-4043-8c56-fb40628ff45a","Type":"ContainerDied","Data":"55762b9b2b0acca1d13951820d5522b2041bb590a24035823c1ff5b3fada9faf"}
Oct 30 22:34:10.836702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:10.836655    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:10.988066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:10.988022    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:11.262855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:11.262772    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:11.276300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:11.276261    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"]
Oct 30 22:34:11.276537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:11.276509    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" containerID="cri-o://32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb" gracePeriod=30
Oct 30 22:34:11.417050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:11.416967    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:12.030253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.030204    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:12.152532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.151221    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca\") pod \"b254728e-4059-4043-8c56-fb40628ff45a\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") "
Oct 30 22:34:12.152532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.151270    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca\") pod \"b254728e-4059-4043-8c56-fb40628ff45a\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") "
Oct 30 22:34:12.152532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.151310    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t\" (UniqueName: \"kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t\") pod \"b254728e-4059-4043-8c56-fb40628ff45a\" (UID: \"b254728e-4059-4043-8c56-fb40628ff45a\") "
Oct 30 22:34:12.152532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.151559    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca" pod "b254728e-4059-4043-8c56-fb40628ff45a" (UID: "b254728e-4059-4043-8c56-fb40628ff45a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:12.152532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.152382    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:12.154768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.154446    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t" pod "b254728e-4059-4043-8c56-fb40628ff45a" (UID: "b254728e-4059-4043-8c56-fb40628ff45a"). InnerVolumeSpecName "kube-api-access-2gt6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:12.154768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.154754    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca" pod "b254728e-4059-4043-8c56-fb40628ff45a" (UID: "b254728e-4059-4043-8c56-fb40628ff45a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:12.254477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.254439    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b254728e-4059-4043-8c56-fb40628ff45a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:12.254477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.254472    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gt6t\" (UniqueName: \"kubernetes.io/projected/b254728e-4059-4043-8c56-fb40628ff45a-kube-api-access-2gt6t\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:12.817563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.817531    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"
Oct 30 22:34:12.817797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.817530    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg" event={"ID":"b254728e-4059-4043-8c56-fb40628ff45a","Type":"ContainerDied","Data":"5cbe4bd40a60132b0109a472b3a024c384ff5d1a6ebad2ca881c032b561082e5"}
Oct 30 22:34:12.817797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:12.817662    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbe4bd40a60132b0109a472b3a024c384ff5d1a6ebad2ca881c032b561082e5"
Oct 30 22:34:13.831633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:13.831537    5028 generic.go:356] "Generic (PLEG): container finished" podID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerID="bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33" exitCode=0
Oct 30 22:34:13.831633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:13.831618    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" event={"ID":"74b31fa1-9460-47b4-bddc-080508b9c5b7","Type":"ContainerDied","Data":"bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"}
Oct 30 22:34:13.832258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:13.832212    5028 scope.go:122] "RemoveContainer" containerID="bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"
Oct 30 22:34:13.834197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:13.834157    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:13.834326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:13.834226    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:13.834326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:13.834258    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:14.852128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:14.852062    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:34:14.852128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:14.852123    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:34:14.858908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:14.858873    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" event={"ID":"74b31fa1-9460-47b4-bddc-080508b9c5b7","Type":"ContainerStarted","Data":"6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0"}
Oct 30 22:34:15.287008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:15.286974    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:34:15.287196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:15.287028    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:34:15.294150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:15.294123    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:34:16.247933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.247901    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:16.290764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.290727    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:34:16.291050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.291018    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:34:16.704842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.704813    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:34:16.877435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.877343    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.877901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.877875    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.877956    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878074    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878111    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878160    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878194    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878340    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878395    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw\") pod \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\" (UID: \"632d78e1-94e3-4c66-bd57-7ecb64bd0c07\") "
Oct 30 22:34:16.878640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.878610    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:16.880462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.880429    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.880668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.880646    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.880879    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.881284    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.881122    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.881944    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883309    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883333    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883347    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883363    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883376    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883390    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.883793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.883406    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.885455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.885426    5028 generic.go:356] "Generic (PLEG): container finished" podID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerID="32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb" exitCode=0
Oct 30 22:34:16.885569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.885477    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"
Oct 30 22:34:16.885569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.885507    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" event={"ID":"632d78e1-94e3-4c66-bd57-7ecb64bd0c07","Type":"ContainerDied","Data":"32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb"}
Oct 30 22:34:16.885716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.885573    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct" event={"ID":"632d78e1-94e3-4c66-bd57-7ecb64bd0c07","Type":"ContainerDied","Data":"eefc8ec31310f15e1c6fc92c0f2fb0a3e22f203d040ef3756bb3ce7dbb2f4d09"}
Oct 30 22:34:16.885716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.885605    5028 scope.go:122] "RemoveContainer" containerID="32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb"
Oct 30 22:34:16.897303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.897242    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.898854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.898830    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw" pod "632d78e1-94e3-4c66-bd57-7ecb64bd0c07" (UID: "632d78e1-94e3-4c66-bd57-7ecb64bd0c07"). InnerVolumeSpecName "kube-api-access-w2vpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:16.962596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.962568    5028 scope.go:122] "RemoveContainer" containerID="32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb"
Oct 30 22:34:16.963080 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:16.963052    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb\": container with ID starting with 32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb not found: ID does not exist" containerID="32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb"
Oct 30 22:34:16.963260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.963092    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb"} err="failed to get container status \"32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb\": rpc error: code = NotFound desc = could not find container \"32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb\": container with ID starting with 32f4de8001aae560ab0871bd706b13d367b306ad7844aa1fda6971cc95a6eddb not found: ID does not exist"
Oct 30 22:34:16.985717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.984744    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vpw\" (UniqueName: \"kubernetes.io/projected/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-kube-api-access-w2vpw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.985949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:16.985914    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632d78e1-94e3-4c66-bd57-7ecb64bd0c07-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.287325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:17.287284    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:34:17.287861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:17.287360    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:34:17.314597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:17.314566    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"]
Oct 30 22:34:17.337711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:17.336324    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-x2fct"]
Oct 30 22:34:17.337711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:17.336963    5028 scope.go:122] "RemoveContainer" containerID="63e92f8880447ef2cb9b0932c5c0eea02aecea1e05faf5fcb61875b5ad70c7d9"
Oct 30 22:34:17.355535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:17.355507    5028 scope.go:122] "RemoveContainer" containerID="3a180592566367e5b460970a68153a8fa9b81af80461641729d830490fa84377"
Oct 30 22:34:18.275694 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:18.275647    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:18.286964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:18.286932    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:34:18.287169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:18.286982    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:34:19.289240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:19.289201    5028 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:34:19.289845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:19.289254    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:34:19.294137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:19.294108    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" path="/var/lib/kubelet/pods/632d78e1-94e3-4c66-bd57-7ecb64bd0c07/volumes"
Oct 30 22:34:19.783335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:19.783301    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"]
Oct 30 22:34:19.888561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:19.888526    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"]
Oct 30 22:34:20.156454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:20.156355    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:20.307705 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:20.307659    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:34:20.312073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:20.312035    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:34:20.315210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:20.315180    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:34:20.320561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:20.320531    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:21.653576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:21.653528    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:24.783990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.783937    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8" podUID="7a931850-938f-469b-9361-c81359628ee2" containerName="payload-processing" containerID="cri-o://9a64dfde34945437672b9759e6c10e44a288b319f64fc473cf22b8ed4f133f3b" gracePeriod=25
Oct 30 22:34:24.889471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.889402    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7" podUID="828c50d6-ff1a-488a-a07d-f778706b5027" containerName="payload-pre-processing" containerID="cri-o://ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e" gracePeriod=25
Oct 30 22:34:24.961602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.961564    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"]
Oct 30 22:34:24.962728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.962703    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b254728e-4059-4043-8c56-fb40628ff45a" containerName="revoke-keys"
Oct 30 22:34:24.962891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.962739    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b254728e-4059-4043-8c56-fb40628ff45a" containerName="revoke-keys"
Oct 30 22:34:24.962891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.962855    5028 cpu_manager.go:414] "containerMap: removing container" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy"
Oct 30 22:34:24.962891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.962864    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy"
Oct 30 22:34:24.963096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.963080    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="632d78e1-94e3-4c66-bd57-7ecb64bd0c07" containerName="istio-proxy"
Oct 30 22:34:24.963159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.963101    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b254728e-4059-4043-8c56-fb40628ff45a" containerName="revoke-keys"
Oct 30 22:34:24.966362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.966325    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:24.970887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.970293    5028 generic.go:356] "Generic (PLEG): container finished" podID="7a931850-938f-469b-9361-c81359628ee2" containerID="9a64dfde34945437672b9759e6c10e44a288b319f64fc473cf22b8ed4f133f3b" exitCode=0
Oct 30 22:34:24.970887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:24.970549    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8" event={"ID":"7a931850-938f-469b-9361-c81359628ee2","Type":"ContainerDied","Data":"9a64dfde34945437672b9759e6c10e44a288b319f64fc473cf22b8ed4f133f3b"}
Oct 30 22:34:25.029951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.029898    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"]
Oct 30 22:34:25.055806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.055319    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlpd\" (UniqueName: \"kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.055806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.055417    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.055806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.055469    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.066205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.066172    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:25.066376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.066263    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:25.066376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.066272    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:34:25.066491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.066409    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:25.113819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.113789    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:34:25.158007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.157960    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlpd\" (UniqueName: \"kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.158206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.158070    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.158206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.158120    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.159187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.159156    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.170305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.170273    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.214844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.214816    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:34:25.220221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.220175    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlpd\" (UniqueName: \"kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd\") pod \"maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.259680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.259635    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg\" (UniqueName: \"kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg\") pod \"7a931850-938f-469b-9361-c81359628ee2\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") "
Oct 30 22:34:25.259918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.259745    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume\") pod \"7a931850-938f-469b-9361-c81359628ee2\" (UID: \"7a931850-938f-469b-9361-c81359628ee2\") "
Oct 30 22:34:25.261437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.261369    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume" pod "7a931850-938f-469b-9361-c81359628ee2" (UID: "7a931850-938f-469b-9361-c81359628ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:25.263062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.263032    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg" pod "7a931850-938f-469b-9361-c81359628ee2" (UID: "7a931850-938f-469b-9361-c81359628ee2"). InnerVolumeSpecName "kube-api-access-l7lxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:25.291874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.291845    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:25.362500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.362031    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume\") pod \"828c50d6-ff1a-488a-a07d-f778706b5027\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") "
Oct 30 22:34:25.362500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.362129    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m\" (UniqueName: \"kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m\") pod \"828c50d6-ff1a-488a-a07d-f778706b5027\" (UID: \"828c50d6-ff1a-488a-a07d-f778706b5027\") "
Oct 30 22:34:25.363323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.363298    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume" pod "828c50d6-ff1a-488a-a07d-f778706b5027" (UID: "828c50d6-ff1a-488a-a07d-f778706b5027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:25.364222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.364196    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7lxg\" (UniqueName: \"kubernetes.io/projected/7a931850-938f-469b-9361-c81359628ee2-kube-api-access-l7lxg\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:25.364321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.364228    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a931850-938f-469b-9361-c81359628ee2-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:25.364321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.364246    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828c50d6-ff1a-488a-a07d-f778706b5027-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:25.365759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.365654    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m" pod "828c50d6-ff1a-488a-a07d-f778706b5027" (UID: "828c50d6-ff1a-488a-a07d-f778706b5027"). InnerVolumeSpecName "kube-api-access-6mz5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:25.465979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.465943    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mz5m\" (UniqueName: \"kubernetes.io/projected/828c50d6-ff1a-488a-a07d-f778706b5027-kube-api-access-6mz5m\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:25.677085 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:34:25.677055    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d7248f_9dbe_4255_9ae3_e99ec867256f.slice/crio-9be6c7a988c24191d3e03d03816f9d0424b7e5bfff740f34fbb449fead8a3aa6 WatchSource:0}: Error finding container 9be6c7a988c24191d3e03d03816f9d0424b7e5bfff740f34fbb449fead8a3aa6: Status 404 returned error can't find the container with id 9be6c7a988c24191d3e03d03816f9d0424b7e5bfff740f34fbb449fead8a3aa6
Oct 30 22:34:25.678276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.678251    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"]
Oct 30 22:34:25.982102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.982066    5028 generic.go:356] "Generic (PLEG): container finished" podID="828c50d6-ff1a-488a-a07d-f778706b5027" containerID="ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e" exitCode=0
Oct 30 22:34:25.982583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.982109    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"
Oct 30 22:34:25.982583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.982148    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7" event={"ID":"828c50d6-ff1a-488a-a07d-f778706b5027","Type":"ContainerDied","Data":"ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e"}
Oct 30 22:34:25.982583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.982205    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7" event={"ID":"828c50d6-ff1a-488a-a07d-f778706b5027","Type":"ContainerDied","Data":"a6fc2670db92727646b6d047c3225d87437db1edc74d082ffa97bd496d55d98b"}
Oct 30 22:34:25.982583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.982234    5028 scope.go:122] "RemoveContainer" containerID="ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e"
Oct 30 22:34:25.983931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.983911    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9" event={"ID":"46d7248f-9dbe-4255-9ae3-e99ec867256f","Type":"ContainerStarted","Data":"9be6c7a988c24191d3e03d03816f9d0424b7e5bfff740f34fbb449fead8a3aa6"}
Oct 30 22:34:25.985620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.985599    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"
Oct 30 22:34:25.985620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.985612    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8" event={"ID":"7a931850-938f-469b-9361-c81359628ee2","Type":"ContainerDied","Data":"141c96718973d1efda4986fafe92bbae1821fa4a4ed4fc4ab0411601d211414b"}
Oct 30 22:34:25.995288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.995270    5028 scope.go:122] "RemoveContainer" containerID="ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e"
Oct 30 22:34:25.995573 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:25.995552    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e\": container with ID starting with ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e not found: ID does not exist" containerID="ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e"
Oct 30 22:34:25.995657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.995578    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e"} err="failed to get container status \"ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e\": rpc error: code = NotFound desc = could not find container \"ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e\": container with ID starting with ccf7140dc9f9635d1e12796532ee6bb3a043942af6d0b6fb9071c9b8c154e34e not found: ID does not exist"
Oct 30 22:34:25.995657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:25.995595    5028 scope.go:122] "RemoveContainer" containerID="9a64dfde34945437672b9759e6c10e44a288b319f64fc473cf22b8ed4f133f3b"
Oct 30 22:34:26.144553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.144516    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"]
Oct 30 22:34:26.183674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.183636    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-721c75-5f454d877f-hr4x7"]
Oct 30 22:34:26.256351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.256319    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"]
Oct 30 22:34:26.267785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.267749    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-721c75-6b765575f7-zklj8"]
Oct 30 22:34:26.776508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.772349    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:26.999860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.999777    5028 generic.go:356] "Generic (PLEG): container finished" podID="46d7248f-9dbe-4255-9ae3-e99ec867256f" containerID="839201a53e624e6a5a3e118c9faaefc1f599b42c0f2d3632b63ae564c95a7789" exitCode=0
Oct 30 22:34:26.999860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:26.999814    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9" event={"ID":"46d7248f-9dbe-4255-9ae3-e99ec867256f","Type":"ContainerDied","Data":"839201a53e624e6a5a3e118c9faaefc1f599b42c0f2d3632b63ae564c95a7789"}
Oct 30 22:34:27.295504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:27.295426    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a931850-938f-469b-9361-c81359628ee2" path="/var/lib/kubelet/pods/7a931850-938f-469b-9361-c81359628ee2/volumes"
Oct 30 22:34:27.295889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:27.295874    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828c50d6-ff1a-488a-a07d-f778706b5027" path="/var/lib/kubelet/pods/828c50d6-ff1a-488a-a07d-f778706b5027/volumes"
Oct 30 22:34:28.198498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.198464    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:28.308419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.308329    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd\" (UniqueName: \"kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd\") pod \"46d7248f-9dbe-4255-9ae3-e99ec867256f\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") "
Oct 30 22:34:28.308419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.308381    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca\") pod \"46d7248f-9dbe-4255-9ae3-e99ec867256f\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") "
Oct 30 22:34:28.308419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.308414    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca\") pod \"46d7248f-9dbe-4255-9ae3-e99ec867256f\" (UID: \"46d7248f-9dbe-4255-9ae3-e99ec867256f\") "
Oct 30 22:34:28.308857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.308784    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca" pod "46d7248f-9dbe-4255-9ae3-e99ec867256f" (UID: "46d7248f-9dbe-4255-9ae3-e99ec867256f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:28.309045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.309030    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.310611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.310588    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca" pod "46d7248f-9dbe-4255-9ae3-e99ec867256f" (UID: "46d7248f-9dbe-4255-9ae3-e99ec867256f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:28.310743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.310720    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd" pod "46d7248f-9dbe-4255-9ae3-e99ec867256f" (UID: "46d7248f-9dbe-4255-9ae3-e99ec867256f"). InnerVolumeSpecName "kube-api-access-6jlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:28.411192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.411147    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlpd\" (UniqueName: \"kubernetes.io/projected/46d7248f-9dbe-4255-9ae3-e99ec867256f-kube-api-access-6jlpd\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.411192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:28.411189    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d7248f-9dbe-4255-9ae3-e99ec867256f-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.438736 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:28.438681    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:34:29.019229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:29.019200    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"
Oct 30 22:34:29.019452 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:29.019417    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9" event={"ID":"46d7248f-9dbe-4255-9ae3-e99ec867256f","Type":"ContainerDied","Data":"9be6c7a988c24191d3e03d03816f9d0424b7e5bfff740f34fbb449fead8a3aa6"}
Oct 30 22:34:29.019522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:29.019467    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be6c7a988c24191d3e03d03816f9d0424b7e5bfff740f34fbb449fead8a3aa6"
Oct 30 22:34:30.329239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:30.329152    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:30.509177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:30.509140    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.038257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:32.038219    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.330432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:32.330341    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"]
Oct 30 22:34:32.453347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:32.453300    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"]
Oct 30 22:34:32.808723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:32.807477    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:33.020343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:33.020275    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:33.912710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:33.909914    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:33.983856 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:33.983795    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:33.984482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:33.984448    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 30 22:34:33.984981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:33.984954    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:34.026665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.026622    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:34.085844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.085793    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:34.240194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.240148    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:34:34.581473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.581369    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Oct 30 22:34:34.910864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.910766    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:34:34.913020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.911540    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:34:34.984998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.984952    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:34.985807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:34.985781    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:34:35.026048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.026009    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:35.026702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.026653    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:34:35.028273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.028244    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:35.028896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.028875    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:34:35.461232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.432813    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:34:35.584102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.583960    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:35.914261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.914162    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Oct 30 22:34:35.930846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.915901    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:34:35.930846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.915992    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:34:35.930846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.916316    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:34:35.981711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:35.979216    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:36.028008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.027959    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk" podUID="5d7eba18-77c0-4727-8a41-a77ee94779b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:36.030534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.030494    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:36.322950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.321679    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:34:36.436807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.436762    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:36.526036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.515818    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:34:36.533726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.528724    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:34:36.631763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.631647    5028 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 30 22:34:36.644267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.644235    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:34:36.681704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.679277    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk"
Oct 30 22:34:36.831889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:36.831847    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:37.343715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:37.337295    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb" podUID="39d72902-7182-4525-ac29-0000ebfd5d65" containerName="payload-processing" containerID="cri-o://e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4" gracePeriod=25
Oct 30 22:34:37.455830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:37.455582    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96" podUID="096b8c19-2b8c-469b-a00f-8c458fb8fac3" containerName="payload-pre-processing" containerID="cri-o://2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e" gracePeriod=25
Oct 30 22:34:37.911822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:37.911794    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:34:38.036205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.036166    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:34:38.083967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.083932    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume\") pod \"39d72902-7182-4525-ac29-0000ebfd5d65\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") "
Oct 30 22:34:38.084198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.083988    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp\" (UniqueName: \"kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp\") pod \"39d72902-7182-4525-ac29-0000ebfd5d65\" (UID: \"39d72902-7182-4525-ac29-0000ebfd5d65\") "
Oct 30 22:34:38.084470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.084442    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume" pod "39d72902-7182-4525-ac29-0000ebfd5d65" (UID: "39d72902-7182-4525-ac29-0000ebfd5d65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:38.086479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.086451    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp" pod "39d72902-7182-4525-ac29-0000ebfd5d65" (UID: "39d72902-7182-4525-ac29-0000ebfd5d65"). InnerVolumeSpecName "kube-api-access-sxnbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:38.186088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.186055    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume\") pod \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") "
Oct 30 22:34:38.186250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.186098    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv\" (UniqueName: \"kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv\") pod \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\" (UID: \"096b8c19-2b8c-469b-a00f-8c458fb8fac3\") "
Oct 30 22:34:38.186553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.186526    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume" pod "096b8c19-2b8c-469b-a00f-8c458fb8fac3" (UID: "096b8c19-2b8c-469b-a00f-8c458fb8fac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:38.186818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.186795    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d72902-7182-4525-ac29-0000ebfd5d65-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:38.186927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.186822    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxnbp\" (UniqueName: \"kubernetes.io/projected/39d72902-7182-4525-ac29-0000ebfd5d65-kube-api-access-sxnbp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:38.186927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.186841    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096b8c19-2b8c-469b-a00f-8c458fb8fac3-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:38.188505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.188477    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv" pod "096b8c19-2b8c-469b-a00f-8c458fb8fac3" (UID: "096b8c19-2b8c-469b-a00f-8c458fb8fac3"). InnerVolumeSpecName "kube-api-access-bpdvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:38.259988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.259950    5028 generic.go:356] "Generic (PLEG): container finished" podID="39d72902-7182-4525-ac29-0000ebfd5d65" containerID="e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4" exitCode=0
Oct 30 22:34:38.259988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.259993    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"
Oct 30 22:34:38.260274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.260045    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb" event={"ID":"39d72902-7182-4525-ac29-0000ebfd5d65","Type":"ContainerDied","Data":"e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4"}
Oct 30 22:34:38.260274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.260105    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb" event={"ID":"39d72902-7182-4525-ac29-0000ebfd5d65","Type":"ContainerDied","Data":"ee27b45e6c998033139819a20adedf2d8c0c7cfea327610ec4317bbb06e5cc19"}
Oct 30 22:34:38.260274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.260136    5028 scope.go:122] "RemoveContainer" containerID="e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4"
Oct 30 22:34:38.262113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.262074    5028 generic.go:356] "Generic (PLEG): container finished" podID="096b8c19-2b8c-469b-a00f-8c458fb8fac3" containerID="2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e" exitCode=0
Oct 30 22:34:38.262248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.262138    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"
Oct 30 22:34:38.262342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.262310    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96" event={"ID":"096b8c19-2b8c-469b-a00f-8c458fb8fac3","Type":"ContainerDied","Data":"2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e"}
Oct 30 22:34:38.262477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.262360    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96" event={"ID":"096b8c19-2b8c-469b-a00f-8c458fb8fac3","Type":"ContainerDied","Data":"460e83207cf9f390ea6f56000998e0098d33bffdbf2a8b5b71eb77480160925d"}
Oct 30 22:34:38.274847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.274824    5028 scope.go:122] "RemoveContainer" containerID="e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4"
Oct 30 22:34:38.275150 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:38.275127    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4\": container with ID starting with e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4 not found: ID does not exist" containerID="e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4"
Oct 30 22:34:38.275229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.275158    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4"} err="failed to get container status \"e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4\": rpc error: code = NotFound desc = could not find container \"e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4\": container with ID starting with e04bb0985d86a74f098d6197dc278540a9ffb35c823d9f68fcd70f7ccb04a8e4 not found: ID does not exist"
Oct 30 22:34:38.275229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.275182    5028 scope.go:122] "RemoveContainer" containerID="2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e"
Oct 30 22:34:38.288269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.288247    5028 scope.go:122] "RemoveContainer" containerID="2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e"
Oct 30 22:34:38.288560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:38.288541    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e\": container with ID starting with 2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e not found: ID does not exist" containerID="2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e"
Oct 30 22:34:38.288642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.288567    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e"} err="failed to get container status \"2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e\": rpc error: code = NotFound desc = could not find container \"2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e\": container with ID starting with 2b4e9d38ed8c30472592f6f6b705124fc4edfd5caee5696d9e7cf2c8f771904e not found: ID does not exist"
Oct 30 22:34:38.288865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.288842    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdvv\" (UniqueName: \"kubernetes.io/projected/096b8c19-2b8c-469b-a00f-8c458fb8fac3-kube-api-access-bpdvv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:38.426682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.426602    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"]
Oct 30 22:34:38.444726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.444228    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c3c2cf-7dc6ff5bb7-9jv96"]
Oct 30 22:34:38.564334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.564301    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"]
Oct 30 22:34:38.574394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.574365    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c3c2cf-6887cd5b5-sghhb"]
Oct 30 22:34:38.696323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.696235    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:38.888032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:38.887997    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:39.171668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.171632    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:39.294930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.294895    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096b8c19-2b8c-469b-a00f-8c458fb8fac3" path="/var/lib/kubelet/pods/096b8c19-2b8c-469b-a00f-8c458fb8fac3/volumes"
Oct 30 22:34:39.295352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.295339    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d72902-7182-4525-ac29-0000ebfd5d65" path="/var/lib/kubelet/pods/39d72902-7182-4525-ac29-0000ebfd5d65/volumes"
Oct 30 22:34:39.409728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.409659    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"]
Oct 30 22:34:39.458490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.458451    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-qxb5m"]
Oct 30 22:34:39.784537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.784449    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"]
Oct 30 22:34:39.785281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785255    5028 cpu_manager.go:414] "containerMap: removing container" podUID="828c50d6-ff1a-488a-a07d-f778706b5027" containerName="payload-pre-processing"
Oct 30 22:34:39.785281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785276    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828c50d6-ff1a-488a-a07d-f778706b5027" containerName="payload-pre-processing"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785291    5028 cpu_manager.go:414] "containerMap: removing container" podUID="39d72902-7182-4525-ac29-0000ebfd5d65" containerName="payload-processing"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785297    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d72902-7182-4525-ac29-0000ebfd5d65" containerName="payload-processing"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785332    5028 cpu_manager.go:414] "containerMap: removing container" podUID="46d7248f-9dbe-4255-9ae3-e99ec867256f" containerName="revoke-keys"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785339    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d7248f-9dbe-4255-9ae3-e99ec867256f" containerName="revoke-keys"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785350    5028 cpu_manager.go:414] "containerMap: removing container" podUID="096b8c19-2b8c-469b-a00f-8c458fb8fac3" containerName="payload-pre-processing"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785357    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096b8c19-2b8c-469b-a00f-8c458fb8fac3" containerName="payload-pre-processing"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785382    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7a931850-938f-469b-9361-c81359628ee2" containerName="payload-processing"
Oct 30 22:34:39.785490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785387    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a931850-938f-469b-9361-c81359628ee2" containerName="payload-processing"
Oct 30 22:34:39.785886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785516    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d7248f-9dbe-4255-9ae3-e99ec867256f" containerName="revoke-keys"
Oct 30 22:34:39.785886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785525    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="096b8c19-2b8c-469b-a00f-8c458fb8fac3" containerName="payload-pre-processing"
Oct 30 22:34:39.785886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785536    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a931850-938f-469b-9361-c81359628ee2" containerName="payload-processing"
Oct 30 22:34:39.785886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785547    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="828c50d6-ff1a-488a-a07d-f778706b5027" containerName="payload-pre-processing"
Oct 30 22:34:39.785886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.785555    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d72902-7182-4525-ac29-0000ebfd5d65" containerName="payload-processing"
Oct 30 22:34:39.791224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.791201    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:39.842418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.842378    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:39.842418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.842386    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:39.849779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.848840    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"]
Oct 30 22:34:39.894432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.894394    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:39.894589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.894497    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:34:39.912768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.912731    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwntx\" (UniqueName: \"kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:39.912941 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.912855    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:39.912941 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:39.912914    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.014811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.014773    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntx\" (UniqueName: \"kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.015027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.014902    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.015027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.014975    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.015727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.015679    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.017702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.017663    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.074906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.074829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwntx\" (UniqueName: \"kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.109223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.109190    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:40.495172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.495139    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"]
Oct 30 22:34:40.495651 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:34:40.495437    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ced388_6a0b_48f9_8924_7e9aae2ce03c.slice/crio-7c56cb81a417fdf37d9ceab1f565d06b11f956226bec645d6f58ffb0275934dd WatchSource:0}: Error finding container 7c56cb81a417fdf37d9ceab1f565d06b11f956226bec645d6f58ffb0275934dd: Status 404 returned error can't find the container with id 7c56cb81a417fdf37d9ceab1f565d06b11f956226bec645d6f58ffb0275934dd
Oct 30 22:34:40.627884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.627844    5028 patch_prober.go:29] interesting pod/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:40.628111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:40.627920    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:41.293877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:41.293846    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450333de-01c3-4fb2-a806-9f9f1ea43fb7" path="/var/lib/kubelet/pods/450333de-01c3-4fb2-a806-9f9f1ea43fb7/volumes"
Oct 30 22:34:41.295140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:41.295105    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d" event={"ID":"b6ced388-6a0b-48f9-8924-7e9aae2ce03c","Type":"ContainerStarted","Data":"c2af00d1a3252ca33efd9daced6381a2bc6f961cebc87faae9bb79ab8a0d1034"}
Oct 30 22:34:41.295296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:41.295151    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d" event={"ID":"b6ced388-6a0b-48f9-8924-7e9aae2ce03c","Type":"ContainerStarted","Data":"7c56cb81a417fdf37d9ceab1f565d06b11f956226bec645d6f58ffb0275934dd"}
Oct 30 22:34:41.417039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:41.416970    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d" podStartSLOduration=2.416952206 podStartE2EDuration="2.416952206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:41.410617274 +0000 UTC m=+2012.260130253" watchObservedRunningTime="2026-10-30 22:34:41.416952206 +0000 UTC m=+2012.266465198"
Oct 30 22:34:42.306087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.306053    5028 generic.go:356] "Generic (PLEG): container finished" podID="b6ced388-6a0b-48f9-8924-7e9aae2ce03c" containerID="c2af00d1a3252ca33efd9daced6381a2bc6f961cebc87faae9bb79ab8a0d1034" exitCode=0
Oct 30 22:34:42.306610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.306135    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d" event={"ID":"b6ced388-6a0b-48f9-8924-7e9aae2ce03c","Type":"ContainerDied","Data":"c2af00d1a3252ca33efd9daced6381a2bc6f961cebc87faae9bb79ab8a0d1034"}
Oct 30 22:34:42.308387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.308358    5028 generic.go:356] "Generic (PLEG): container finished" podID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerID="e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341" exitCode=0
Oct 30 22:34:42.308520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.308438    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" event={"ID":"bcbea46f-84fc-4ea1-9c67-e395f6bc0748","Type":"ContainerDied","Data":"e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"}
Oct 30 22:34:42.308930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.308915    5028 scope.go:122] "RemoveContainer" containerID="e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"
Oct 30 22:34:42.310823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.310789    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:42.310934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.310877    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:42.310934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:42.310925    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:34:43.332710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.330798    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" event={"ID":"bcbea46f-84fc-4ea1-9c67-e395f6bc0748","Type":"ContainerStarted","Data":"4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356"}
Oct 30 22:34:43.568362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.568335    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:43.691573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.691487    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca\") pod \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") "
Oct 30 22:34:43.691573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.691528    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx\" (UniqueName: \"kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx\") pod \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") "
Oct 30 22:34:43.691844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.691746    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca\") pod \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\" (UID: \"b6ced388-6a0b-48f9-8924-7e9aae2ce03c\") "
Oct 30 22:34:43.691912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.691829    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca" pod "b6ced388-6a0b-48f9-8924-7e9aae2ce03c" (UID: "b6ced388-6a0b-48f9-8924-7e9aae2ce03c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:43.692248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.692229    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.693815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.693791    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca" pod "b6ced388-6a0b-48f9-8924-7e9aae2ce03c" (UID: "b6ced388-6a0b-48f9-8924-7e9aae2ce03c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:43.693914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.693866    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx" pod "b6ced388-6a0b-48f9-8924-7e9aae2ce03c" (UID: "b6ced388-6a0b-48f9-8924-7e9aae2ce03c"). InnerVolumeSpecName "kube-api-access-vwntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:43.700004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.699981    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:34:43.700211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.700192    5028 patch_prober.go:29] interesting pod/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:34:43.700289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.700244    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 22:34:43.793756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.793678    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwntx\" (UniqueName: \"kubernetes.io/projected/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-kube-api-access-vwntx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.793756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.793752    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ced388-6a0b-48f9-8924-7e9aae2ce03c-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.907512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:43.906102    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.114876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:44.114798    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.353278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:44.353231    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"
Oct 30 22:34:44.353710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:44.353257    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d" event={"ID":"b6ced388-6a0b-48f9-8924-7e9aae2ce03c","Type":"ContainerDied","Data":"7c56cb81a417fdf37d9ceab1f565d06b11f956226bec645d6f58ffb0275934dd"}
Oct 30 22:34:44.353710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:44.353325    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c56cb81a417fdf37d9ceab1f565d06b11f956226bec645d6f58ffb0275934dd"
Oct 30 22:34:44.700670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:44.700634    5028 patch_prober.go:29] interesting pod/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:34:44.700874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:44.700725    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 22:34:45.701002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:45.700968    5028 patch_prober.go:29] interesting pod/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:34:45.701436 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:45.701028    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 22:34:46.700844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:46.700804    5028 patch_prober.go:29] interesting pod/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:34:46.701032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:46.700882    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 22:34:47.700976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:47.700934    5028 patch_prober.go:29] interesting pod/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:34:47.701179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:47.700995    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 30 22:34:47.749805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:47.746302    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:47.897402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:47.897325    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:48.775942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:48.775907    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:34:48.778506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:48.778126    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:34:48.779030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:48.779006    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:34:48.968255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:48.968222    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:50.069595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.069560    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:50.398008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.397908    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:50.517311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.516578    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"]
Oct 30 22:34:50.517516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.517446    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b6ced388-6a0b-48f9-8924-7e9aae2ce03c" containerName="revoke-keys"
Oct 30 22:34:50.517516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.517461    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ced388-6a0b-48f9-8924-7e9aae2ce03c" containerName="revoke-keys"
Oct 30 22:34:50.517648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.517601    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ced388-6a0b-48f9-8924-7e9aae2ce03c" containerName="revoke-keys"
Oct 30 22:34:50.524615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.524591    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.580714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.580663    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"]
Oct 30 22:34:50.596429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.596223    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb2b\" (UniqueName: \"kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.597734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.597705    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.622699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.622655    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-29bbe735\""
Oct 30 22:34:50.622897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.622847    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-29bbe735-dockercfg-nq2bf\""
Oct 30 22:34:50.710243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.701723    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb2b\" (UniqueName: \"kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.754805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.750372    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.754805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.751675    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.822345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.818735    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"]
Oct 30 22:34:50.824893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.824825    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb2b\" (UniqueName: \"kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b\") pod \"payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.852714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.849435    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:50.852714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.850546    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:50.897150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.897120    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"]
Oct 30 22:34:50.903499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.903380    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"]
Oct 30 22:34:50.920763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.909883    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:50.920763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.912558    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"]
Oct 30 22:34:50.957766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.957736    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:50.960189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.958585    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6p9m\" (UniqueName: \"kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:50.960425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.960375    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:34:50.968500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.968464    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:50.979411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.979387    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:50.979650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:50.979636    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:51.060716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.059965    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6cg\" (UniqueName: \"kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.060716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.060168    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6p9m\" (UniqueName: \"kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:51.060716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.060253    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.060716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.060347    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:51.060716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.060401    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.061456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.061423    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:51.120840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.120806    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6p9m\" (UniqueName: \"kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m\") pod \"payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") " pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:51.161349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.161287    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.161555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.161433    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.161626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.161609    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6cg\" (UniqueName: \"kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.162262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.162231    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.163739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.163717    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.201612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.201583    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:51.222208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.222173    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6cg\" (UniqueName: \"kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.267204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.266559    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:51.326091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.326056    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"]
Oct 30 22:34:51.467807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.467759    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz" event={"ID":"00b20f1e-4231-4cf8-9cdb-2f1191307323","Type":"ContainerStarted","Data":"400e3a56744323223c2cb0e2d190126f3e1f1aa212e0c0d24f7948f846698195"}
Oct 30 22:34:51.467969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.467867    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:34:51.582927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.582830    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz" podStartSLOduration=1.58281643 podStartE2EDuration="1.58281643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:51.581732671 +0000 UTC m=+2022.431245642" watchObservedRunningTime="2026-10-30 22:34:51.58281643 +0000 UTC m=+2022.432329401"
Oct 30 22:34:51.590311 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:34:51.590278    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa019642_4b4e_43fb_9b47_f83f3744f656.slice/crio-e8f7162594d80d9dc466796b60ac1fc5023a40bea50e701ff9208e7f9b32458c WatchSource:0}: Error finding container e8f7162594d80d9dc466796b60ac1fc5023a40bea50e701ff9208e7f9b32458c: Status 404 returned error can't find the container with id e8f7162594d80d9dc466796b60ac1fc5023a40bea50e701ff9208e7f9b32458c
Oct 30 22:34:51.593490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.593461    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"]
Oct 30 22:34:51.684993 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:34:51.684966    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73b7ad3_38a8_4b2d_a3be_447c13fffc94.slice/crio-173012abcabbefa5d6b6a9f29de6deae1c24515adf13479fdbb53499abf55779 WatchSource:0}: Error finding container 173012abcabbefa5d6b6a9f29de6deae1c24515adf13479fdbb53499abf55779: Status 404 returned error can't find the container with id 173012abcabbefa5d6b6a9f29de6deae1c24515adf13479fdbb53499abf55779
Oct 30 22:34:51.685342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:51.685320    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"]
Oct 30 22:34:52.387825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.387743    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"]
Oct 30 22:34:52.479582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.479547    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz" event={"ID":"00b20f1e-4231-4cf8-9cdb-2f1191307323","Type":"ContainerStarted","Data":"fb30449ede5f938f5db3fd814838de79274522087ed7a46414f247d75f08350f"}
Oct 30 22:34:52.481585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.481555    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn" event={"ID":"b73b7ad3-38a8-4b2d-a3be-447c13fffc94","Type":"ContainerStarted","Data":"d9016e1bfa9a5e84a9d4dd38fadb6d734199506545875427f2b796634db0c50a"}
Oct 30 22:34:52.481744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.481602    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn" event={"ID":"b73b7ad3-38a8-4b2d-a3be-447c13fffc94","Type":"ContainerStarted","Data":"173012abcabbefa5d6b6a9f29de6deae1c24515adf13479fdbb53499abf55779"}
Oct 30 22:34:52.483859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.483837    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n" event={"ID":"aa019642-4b4e-43fb-9b47-f83f3744f656","Type":"ContainerStarted","Data":"a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853"}
Oct 30 22:34:52.483859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.483869    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n" event={"ID":"aa019642-4b4e-43fb-9b47-f83f3744f656","Type":"ContainerStarted","Data":"e8f7162594d80d9dc466796b60ac1fc5023a40bea50e701ff9208e7f9b32458c"}
Oct 30 22:34:52.484027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.483899    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:34:52.484354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.484333    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"]
Oct 30 22:34:52.627937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.627873    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn" podStartSLOduration=2.627854633 podStartE2EDuration="2.627854633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:52.622652882 +0000 UTC m=+2023.472165853" watchObservedRunningTime="2026-10-30 22:34:52.627854633 +0000 UTC m=+2023.477367607"
Oct 30 22:34:52.758225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.758190    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:52.853511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.853441    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n" podStartSLOduration=2.853422406 podStartE2EDuration="2.853422406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:52.848943899 +0000 UTC m=+2023.698456873" watchObservedRunningTime="2026-10-30 22:34:52.853422406 +0000 UTC m=+2023.702935378"
Oct 30 22:34:52.927937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:52.927895    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:53.495431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:53.495394    5028 generic.go:356] "Generic (PLEG): container finished" podID="b73b7ad3-38a8-4b2d-a3be-447c13fffc94" containerID="d9016e1bfa9a5e84a9d4dd38fadb6d734199506545875427f2b796634db0c50a" exitCode=0
Oct 30 22:34:53.495958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:53.495474    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn" event={"ID":"b73b7ad3-38a8-4b2d-a3be-447c13fffc94","Type":"ContainerDied","Data":"d9016e1bfa9a5e84a9d4dd38fadb6d734199506545875427f2b796634db0c50a"}
Oct 30 22:34:54.025772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.025723    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:54.701330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.701304    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:54.774102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.774010    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca\") pod \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") "
Oct 30 22:34:54.774265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.774138    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca\") pod \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") "
Oct 30 22:34:54.774345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.774265    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg\" (UniqueName: \"kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg\") pod \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\" (UID: \"b73b7ad3-38a8-4b2d-a3be-447c13fffc94\") "
Oct 30 22:34:54.774502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.774473    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca" pod "b73b7ad3-38a8-4b2d-a3be-447c13fffc94" (UID: "b73b7ad3-38a8-4b2d-a3be-447c13fffc94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:54.774929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.774911    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:54.776254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.776234    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca" pod "b73b7ad3-38a8-4b2d-a3be-447c13fffc94" (UID: "b73b7ad3-38a8-4b2d-a3be-447c13fffc94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:54.776490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.776460    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg" pod "b73b7ad3-38a8-4b2d-a3be-447c13fffc94" (UID: "b73b7ad3-38a8-4b2d-a3be-447c13fffc94"). InnerVolumeSpecName "kube-api-access-jd6cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:54.876511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.876458    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:54.876511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:54.876505    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd6cg\" (UniqueName: \"kubernetes.io/projected/b73b7ad3-38a8-4b2d-a3be-447c13fffc94-kube-api-access-jd6cg\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:55.513980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:55.513951    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"
Oct 30 22:34:55.514165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:55.513989    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn" event={"ID":"b73b7ad3-38a8-4b2d-a3be-447c13fffc94","Type":"ContainerDied","Data":"173012abcabbefa5d6b6a9f29de6deae1c24515adf13479fdbb53499abf55779"}
Oct 30 22:34:55.514165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:55.514036    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="173012abcabbefa5d6b6a9f29de6deae1c24515adf13479fdbb53499abf55779"
Oct 30 22:34:56.175009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.174975    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"]
Oct 30 22:34:56.175803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.175782    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b73b7ad3-38a8-4b2d-a3be-447c13fffc94" containerName="revoke-keys"
Oct 30 22:34:56.175803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.175800    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73b7ad3-38a8-4b2d-a3be-447c13fffc94" containerName="revoke-keys"
Oct 30 22:34:56.175999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.175963    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73b7ad3-38a8-4b2d-a3be-447c13fffc94" containerName="revoke-keys"
Oct 30 22:34:56.178946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.178925    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.231899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.231864    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:56.232071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.231924    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:56.279407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.279373    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:56.279407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.279391    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:34:56.284521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.284492    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"]
Oct 30 22:34:56.297882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.297850    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.298041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.297986    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdqmz\" (UniqueName: \"kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.298084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.298050    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.399234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.399192    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.399437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.399287    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdqmz\" (UniqueName: \"kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.399437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.399350    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.399916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.399895    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.401881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.401862    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.459158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.459119    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdqmz\" (UniqueName: \"kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz\") pod \"maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.494487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.494453    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:56.885561 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:34:56.885526    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570f3dab_f32a_46b4_a89c_bec3615181d5.slice/crio-6cbbf9dfac91a651b0510bf2d69c0f7ba8ecf63c2a30e85d3b9e356f2fb9310a WatchSource:0}: Error finding container 6cbbf9dfac91a651b0510bf2d69c0f7ba8ecf63c2a30e85d3b9e356f2fb9310a: Status 404 returned error can't find the container with id 6cbbf9dfac91a651b0510bf2d69c0f7ba8ecf63c2a30e85d3b9e356f2fb9310a
Oct 30 22:34:56.893549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:56.893526    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"]
Oct 30 22:34:57.388998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.388941    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2" podUID="0715e293-39d3-4b44-8429-53ab9827126e" containerName="payload-processing" containerID="cri-o://c6ad0321a6608f7e0319230c3376b29747d85247745575c5866fe61463faca65" gracePeriod=25
Oct 30 22:34:57.485088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.485014    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn" podUID="c2194013-886a-48da-992e-d00192443e1f" containerName="payload-pre-processing" containerID="cri-o://66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e" gracePeriod=25
Oct 30 22:34:57.538967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.538931    5028 generic.go:356] "Generic (PLEG): container finished" podID="0715e293-39d3-4b44-8429-53ab9827126e" containerID="c6ad0321a6608f7e0319230c3376b29747d85247745575c5866fe61463faca65" exitCode=0
Oct 30 22:34:57.539130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.539009    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2" event={"ID":"0715e293-39d3-4b44-8429-53ab9827126e","Type":"ContainerDied","Data":"c6ad0321a6608f7e0319230c3376b29747d85247745575c5866fe61463faca65"}
Oct 30 22:34:57.542471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.542441    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf" event={"ID":"570f3dab-f32a-46b4-a89c-bec3615181d5","Type":"ContainerStarted","Data":"d317f26fcc8a2abd58dd4ad4f24d4ceb211e7c3ec9838ab01f6c937937d1fc9b"}
Oct 30 22:34:57.542633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.542500    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf" event={"ID":"570f3dab-f32a-46b4-a89c-bec3615181d5","Type":"ContainerStarted","Data":"6cbbf9dfac91a651b0510bf2d69c0f7ba8ecf63c2a30e85d3b9e356f2fb9310a"}
Oct 30 22:34:57.677314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.662611    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf" podStartSLOduration=1.662600631 podStartE2EDuration="1.662600631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:57.662262385 +0000 UTC m=+2028.511775359" watchObservedRunningTime="2026-10-30 22:34:57.662600631 +0000 UTC m=+2028.512113601"
Oct 30 22:34:57.825779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.824657    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:34:57.861381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.861353    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:34:57.923229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.923198    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8\" (UniqueName: \"kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8\") pod \"c2194013-886a-48da-992e-d00192443e1f\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") "
Oct 30 22:34:57.923376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.923361    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume\") pod \"c2194013-886a-48da-992e-d00192443e1f\" (UID: \"c2194013-886a-48da-992e-d00192443e1f\") "
Oct 30 22:34:57.923852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.923826    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume" pod "c2194013-886a-48da-992e-d00192443e1f" (UID: "c2194013-886a-48da-992e-d00192443e1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:57.924177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.924156    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2194013-886a-48da-992e-d00192443e1f-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:57.927158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:57.927129    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8" pod "c2194013-886a-48da-992e-d00192443e1f" (UID: "c2194013-886a-48da-992e-d00192443e1f"). InnerVolumeSpecName "kube-api-access-5j8t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:58.025040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.024961    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl\" (UniqueName: \"kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl\") pod \"0715e293-39d3-4b44-8429-53ab9827126e\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") "
Oct 30 22:34:58.025196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.025110    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume\") pod \"0715e293-39d3-4b44-8429-53ab9827126e\" (UID: \"0715e293-39d3-4b44-8429-53ab9827126e\") "
Oct 30 22:34:58.025539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.025506    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume" pod "0715e293-39d3-4b44-8429-53ab9827126e" (UID: "0715e293-39d3-4b44-8429-53ab9827126e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:58.025927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.025901    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0715e293-39d3-4b44-8429-53ab9827126e-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:58.026021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.025944    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8t8\" (UniqueName: \"kubernetes.io/projected/c2194013-886a-48da-992e-d00192443e1f-kube-api-access-5j8t8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:58.027832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.027808    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl" pod "0715e293-39d3-4b44-8429-53ab9827126e" (UID: "0715e293-39d3-4b44-8429-53ab9827126e"). InnerVolumeSpecName "kube-api-access-v7nxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:58.128227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.128186    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nxl\" (UniqueName: \"kubernetes.io/projected/0715e293-39d3-4b44-8429-53ab9827126e-kube-api-access-v7nxl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:58.515276 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:58.515234    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:34:58.607304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.607276    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"
Oct 30 22:34:58.607304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.607296    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2" event={"ID":"0715e293-39d3-4b44-8429-53ab9827126e","Type":"ContainerDied","Data":"f1b18e3fa82416528b655525ab29548749959bdc503fe4915e8d4b7b52ed2e23"}
Oct 30 22:34:58.607567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.607382    5028 scope.go:122] "RemoveContainer" containerID="c6ad0321a6608f7e0319230c3376b29747d85247745575c5866fe61463faca65"
Oct 30 22:34:58.609338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.609312    5028 generic.go:356] "Generic (PLEG): container finished" podID="570f3dab-f32a-46b4-a89c-bec3615181d5" containerID="d317f26fcc8a2abd58dd4ad4f24d4ceb211e7c3ec9838ab01f6c937937d1fc9b" exitCode=0
Oct 30 22:34:58.609488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.609434    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf" event={"ID":"570f3dab-f32a-46b4-a89c-bec3615181d5","Type":"ContainerDied","Data":"d317f26fcc8a2abd58dd4ad4f24d4ceb211e7c3ec9838ab01f6c937937d1fc9b"}
Oct 30 22:34:58.611248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.611222    5028 generic.go:356] "Generic (PLEG): container finished" podID="c2194013-886a-48da-992e-d00192443e1f" containerID="66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e" exitCode=0
Oct 30 22:34:58.611364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.611291    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"
Oct 30 22:34:58.611413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.611293    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn" event={"ID":"c2194013-886a-48da-992e-d00192443e1f","Type":"ContainerDied","Data":"66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e"}
Oct 30 22:34:58.611456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.611426    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn" event={"ID":"c2194013-886a-48da-992e-d00192443e1f","Type":"ContainerDied","Data":"aef0557fa02dd34c079296b182c4128a880d7999307b2389ee2cebd347adbcf8"}
Oct 30 22:34:58.620410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.620386    5028 scope.go:122] "RemoveContainer" containerID="66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e"
Oct 30 22:34:58.632289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.632263    5028 scope.go:122] "RemoveContainer" containerID="66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e"
Oct 30 22:34:58.632577 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:34:58.632556    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e\": container with ID starting with 66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e not found: ID does not exist" containerID="66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e"
Oct 30 22:34:58.632647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.632594    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e"} err="failed to get container status \"66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e\": rpc error: code = NotFound desc = could not find container \"66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e\": container with ID starting with 66354a5aa1fa6cb7582c667602f0be13ed6c1e9588ffc2baeece6692f733993e not found: ID does not exist"
Oct 30 22:34:58.773834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.773755    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"]
Oct 30 22:34:58.820510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.820436    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-71c235-64bb4ccc5f-9z2d2"]
Oct 30 22:34:58.995607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:58.995573    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"]
Oct 30 22:34:59.003791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.003761    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-71c235-55565ffff-5zmbn"]
Oct 30 22:34:59.255825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.255783    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:59.321715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.316640    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0715e293-39d3-4b44-8429-53ab9827126e" path="/var/lib/kubelet/pods/0715e293-39d3-4b44-8429-53ab9827126e/volumes"
Oct 30 22:34:59.321715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.317457    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2194013-886a-48da-992e-d00192443e1f" path="/var/lib/kubelet/pods/c2194013-886a-48da-992e-d00192443e1f/volumes"
Oct 30 22:34:59.807370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.807340    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:34:59.959752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.959723    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca\") pod \"570f3dab-f32a-46b4-a89c-bec3615181d5\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") "
Oct 30 22:34:59.959932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.959849    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz\" (UniqueName: \"kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz\") pod \"570f3dab-f32a-46b4-a89c-bec3615181d5\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") "
Oct 30 22:34:59.960012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.959939    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca\") pod \"570f3dab-f32a-46b4-a89c-bec3615181d5\" (UID: \"570f3dab-f32a-46b4-a89c-bec3615181d5\") "
Oct 30 22:34:59.960139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.960116    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca" pod "570f3dab-f32a-46b4-a89c-bec3615181d5" (UID: "570f3dab-f32a-46b4-a89c-bec3615181d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:59.960666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.960633    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.963165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.963139    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz" pod "570f3dab-f32a-46b4-a89c-bec3615181d5" (UID: "570f3dab-f32a-46b4-a89c-bec3615181d5"). InnerVolumeSpecName "kube-api-access-wdqmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:59.963262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:34:59.963222    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca" pod "570f3dab-f32a-46b4-a89c-bec3615181d5" (UID: "570f3dab-f32a-46b4-a89c-bec3615181d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:00.061716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:00.061649    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdqmz\" (UniqueName: \"kubernetes.io/projected/570f3dab-f32a-46b4-a89c-bec3615181d5-kube-api-access-wdqmz\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:00.061716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:00.061683    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570f3dab-f32a-46b4-a89c-bec3615181d5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:00.634553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:00.634519    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"
Oct 30 22:35:00.634553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:00.634541    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf" event={"ID":"570f3dab-f32a-46b4-a89c-bec3615181d5","Type":"ContainerDied","Data":"6cbbf9dfac91a651b0510bf2d69c0f7ba8ecf63c2a30e85d3b9e356f2fb9310a"}
Oct 30 22:35:00.634849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:00.634592    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cbbf9dfac91a651b0510bf2d69c0f7ba8ecf63c2a30e85d3b9e356f2fb9310a"
Oct 30 22:35:03.027627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:03.027591    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"]
Oct 30 22:35:03.292509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:03.292410    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:03.463487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:03.463450    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:03.503550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:03.503520    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:35:03.504057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:03.504030    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:35:04.575880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:04.575831    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:07.639606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:07.639569    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"]
Oct 30 22:35:07.747125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:07.747091    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"]
Oct 30 22:35:08.005527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.005457    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:08.028326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.028285    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz" podUID="64ebb9a4-b0ac-438b-a5ee-c22b618658dd" containerName="payload-pre-processing" containerID="cri-o://2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1" gracePeriod=25
Oct 30 22:35:08.187660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.184209    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:08.376747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.376710    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:35:08.427935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.427896    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd\" (UniqueName: \"kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd\") pod \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") "
Oct 30 22:35:08.428154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.428030    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume\") pod \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\" (UID: \"64ebb9a4-b0ac-438b-a5ee-c22b618658dd\") "
Oct 30 22:35:08.428487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.428460    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume" pod "64ebb9a4-b0ac-438b-a5ee-c22b618658dd" (UID: "64ebb9a4-b0ac-438b-a5ee-c22b618658dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:08.428751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.428730    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.430395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.430368    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd" pod "64ebb9a4-b0ac-438b-a5ee-c22b618658dd" (UID: "64ebb9a4-b0ac-438b-a5ee-c22b618658dd"). InnerVolumeSpecName "kube-api-access-prxzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:08.529780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.529736    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxzd\" (UniqueName: \"kubernetes.io/projected/64ebb9a4-b0ac-438b-a5ee-c22b618658dd-kube-api-access-prxzd\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.744821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.744786    5028 generic.go:356] "Generic (PLEG): container finished" podID="64ebb9a4-b0ac-438b-a5ee-c22b618658dd" containerID="2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1" exitCode=0
Oct 30 22:35:08.745331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.744842    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"
Oct 30 22:35:08.745331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.745066    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz" event={"ID":"64ebb9a4-b0ac-438b-a5ee-c22b618658dd","Type":"ContainerDied","Data":"2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1"}
Oct 30 22:35:08.745331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.745120    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz" event={"ID":"64ebb9a4-b0ac-438b-a5ee-c22b618658dd","Type":"ContainerDied","Data":"6e0625f3b7528c985507cfc416c776e11e238e0f19966a9b28d7140a3b8b1626"}
Oct 30 22:35:08.745331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.745151    5028 scope.go:122] "RemoveContainer" containerID="2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1"
Oct 30 22:35:08.760827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.760806    5028 scope.go:122] "RemoveContainer" containerID="2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1"
Oct 30 22:35:08.761122 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:35:08.761098    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1\": container with ID starting with 2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1 not found: ID does not exist" containerID="2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1"
Oct 30 22:35:08.761219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.761129    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1"} err="failed to get container status \"2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1\": rpc error: code = NotFound desc = could not find container \"2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1\": container with ID starting with 2125dad9ded5302235ffe019d6f8388f57c0a302a0c4ca66d6f56a19ef9cf9f1 not found: ID does not exist"
Oct 30 22:35:08.911074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.911003    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"]
Oct 30 22:35:08.923302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:08.923264    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-73735c-55ff4fb7cf-fwdsz"]
Oct 30 22:35:09.330806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:09.330760    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ebb9a4-b0ac-438b-a5ee-c22b618658dd" path="/var/lib/kubelet/pods/64ebb9a4-b0ac-438b-a5ee-c22b618658dd/volumes"
Oct 30 22:35:09.937730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:09.936195    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.228241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:11.228192    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:11.494379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:11.494301    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:12.640749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:12.640705    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh" podUID="41a6a05e-1d05-4a5d-a336-952dd731df02" containerName="payload-processing" containerID="cri-o://75f1666e5f84d3bc9ba339b75796c10b7ef1051f38f394614453715cab327aa1" gracePeriod=25
Oct 30 22:35:12.747881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:12.747825    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr" podUID="5b248468-49b4-4f41-a435-667efec10460" containerName="payload-pre-processing" containerID="cri-o://3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0" gracePeriod=25
Oct 30 22:35:12.782298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:12.782266    5028 generic.go:356] "Generic (PLEG): container finished" podID="41a6a05e-1d05-4a5d-a336-952dd731df02" containerID="75f1666e5f84d3bc9ba339b75796c10b7ef1051f38f394614453715cab327aa1" exitCode=0
Oct 30 22:35:12.782438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:12.782336    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh" event={"ID":"41a6a05e-1d05-4a5d-a336-952dd731df02","Type":"ContainerDied","Data":"75f1666e5f84d3bc9ba339b75796c10b7ef1051f38f394614453715cab327aa1"}
Oct 30 22:35:12.976579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:12.976545    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:35:13.014223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.014192    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8\" (UniqueName: \"kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8\") pod \"41a6a05e-1d05-4a5d-a336-952dd731df02\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") "
Oct 30 22:35:13.014429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.014289    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume\") pod \"41a6a05e-1d05-4a5d-a336-952dd731df02\" (UID: \"41a6a05e-1d05-4a5d-a336-952dd731df02\") "
Oct 30 22:35:13.014997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.014962    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume" pod "41a6a05e-1d05-4a5d-a336-952dd731df02" (UID: "41a6a05e-1d05-4a5d-a336-952dd731df02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:13.015357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.015335    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6a05e-1d05-4a5d-a336-952dd731df02-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:13.019117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.019087    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8" pod "41a6a05e-1d05-4a5d-a336-952dd731df02" (UID: "41a6a05e-1d05-4a5d-a336-952dd731df02"). InnerVolumeSpecName "kube-api-access-k5pv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:13.048325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.048287    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:35:13.116504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.116473    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume\") pod \"5b248468-49b4-4f41-a435-667efec10460\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") "
Oct 30 22:35:13.116737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.116590    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9\" (UniqueName: \"kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9\") pod \"5b248468-49b4-4f41-a435-667efec10460\" (UID: \"5b248468-49b4-4f41-a435-667efec10460\") "
Oct 30 22:35:13.117046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.117018    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume" pod "5b248468-49b4-4f41-a435-667efec10460" (UID: "5b248468-49b4-4f41-a435-667efec10460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:13.117341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.117321    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5pv8\" (UniqueName: \"kubernetes.io/projected/41a6a05e-1d05-4a5d-a336-952dd731df02-kube-api-access-k5pv8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:13.117423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.117344    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b248468-49b4-4f41-a435-667efec10460-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:13.119621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.119597    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9" pod "5b248468-49b4-4f41-a435-667efec10460" (UID: "5b248468-49b4-4f41-a435-667efec10460"). InnerVolumeSpecName "kube-api-access-48gh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:13.219192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.219149    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gh9\" (UniqueName: \"kubernetes.io/projected/5b248468-49b4-4f41-a435-667efec10460-kube-api-access-48gh9\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:13.795434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.795391    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh" event={"ID":"41a6a05e-1d05-4a5d-a336-952dd731df02","Type":"ContainerDied","Data":"cc5916220c002ba50ce18bc09723df2bf88b35a27b7016ee5bfab80a0a76efa9"}
Oct 30 22:35:13.795434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.795428    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"
Oct 30 22:35:13.796010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.795509    5028 scope.go:122] "RemoveContainer" containerID="75f1666e5f84d3bc9ba339b75796c10b7ef1051f38f394614453715cab327aa1"
Oct 30 22:35:13.797017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.796994    5028 generic.go:356] "Generic (PLEG): container finished" podID="5b248468-49b4-4f41-a435-667efec10460" containerID="3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0" exitCode=0
Oct 30 22:35:13.797097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.797061    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"
Oct 30 22:35:13.797153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.797081    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr" event={"ID":"5b248468-49b4-4f41-a435-667efec10460","Type":"ContainerDied","Data":"3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0"}
Oct 30 22:35:13.797153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.797138    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr" event={"ID":"5b248468-49b4-4f41-a435-667efec10460","Type":"ContainerDied","Data":"c98c22d3833098eb262290db13677ff6d57998aefe8241e7611801c0053a5daf"}
Oct 30 22:35:13.807802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.807779    5028 scope.go:122] "RemoveContainer" containerID="3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0"
Oct 30 22:35:13.818924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.818905    5028 scope.go:122] "RemoveContainer" containerID="3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0"
Oct 30 22:35:13.819231 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:35:13.819207    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0\": container with ID starting with 3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0 not found: ID does not exist" containerID="3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0"
Oct 30 22:35:13.819318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.819242    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0"} err="failed to get container status \"3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0\": rpc error: code = NotFound desc = could not find container \"3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0\": container with ID starting with 3e1b089ad264785a7d6446927fb082af4803e891a8b114acf5e544cf55a9d2b0 not found: ID does not exist"
Oct 30 22:35:13.973102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:13.973064    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"]
Oct 30 22:35:14.019774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:14.019739    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a3600a-5767d5467c-snzfh"]
Oct 30 22:35:14.143784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:14.143744    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"]
Oct 30 22:35:14.192278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:14.192239    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a3600a-684485b7f9-7pnqr"]
Oct 30 22:35:14.982981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:14.982948    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:15.295171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:15.295094    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a6a05e-1d05-4a5d-a336-952dd731df02" path="/var/lib/kubelet/pods/41a6a05e-1d05-4a5d-a336-952dd731df02/volumes"
Oct 30 22:35:15.295622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:15.295608    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b248468-49b4-4f41-a435-667efec10460" path="/var/lib/kubelet/pods/5b248468-49b4-4f41-a435-667efec10460/volumes"
Oct 30 22:35:20.510204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:20.510150    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:25.318094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:25.318051    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:28.437998 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:35:28.437961    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:35:30.038237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:30.038194    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:32.063311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:32.063272    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:32.288394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:32.288361    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:34.462746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:34.462709    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:38.988892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:38.988856    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:41.835629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:41.835593    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:42.582026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.581957    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"]
Oct 30 22:35:42.583497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.583466    5028 cpu_manager.go:414] "containerMap: removing container" podUID="64ebb9a4-b0ac-438b-a5ee-c22b618658dd" containerName="payload-pre-processing"
Oct 30 22:35:42.583669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.583654    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ebb9a4-b0ac-438b-a5ee-c22b618658dd" containerName="payload-pre-processing"
Oct 30 22:35:42.583867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.583848    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c2194013-886a-48da-992e-d00192443e1f" containerName="payload-pre-processing"
Oct 30 22:35:42.583985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.583970    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2194013-886a-48da-992e-d00192443e1f" containerName="payload-pre-processing"
Oct 30 22:35:42.584147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584135    5028 cpu_manager.go:414] "containerMap: removing container" podUID="570f3dab-f32a-46b4-a89c-bec3615181d5" containerName="revoke-keys"
Oct 30 22:35:42.584229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584218    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570f3dab-f32a-46b4-a89c-bec3615181d5" containerName="revoke-keys"
Oct 30 22:35:42.584308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584299    5028 cpu_manager.go:414] "containerMap: removing container" podUID="41a6a05e-1d05-4a5d-a336-952dd731df02" containerName="payload-processing"
Oct 30 22:35:42.584389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584379    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a6a05e-1d05-4a5d-a336-952dd731df02" containerName="payload-processing"
Oct 30 22:35:42.584483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584474    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5b248468-49b4-4f41-a435-667efec10460" containerName="payload-pre-processing"
Oct 30 22:35:42.584563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584552    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b248468-49b4-4f41-a435-667efec10460" containerName="payload-pre-processing"
Oct 30 22:35:42.584674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584663    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0715e293-39d3-4b44-8429-53ab9827126e" containerName="payload-processing"
Oct 30 22:35:42.584788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.584772    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0715e293-39d3-4b44-8429-53ab9827126e" containerName="payload-processing"
Oct 30 22:35:42.585068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.585051    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="570f3dab-f32a-46b4-a89c-bec3615181d5" containerName="revoke-keys"
Oct 30 22:35:42.585166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.585156    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a6a05e-1d05-4a5d-a336-952dd731df02" containerName="payload-processing"
Oct 30 22:35:42.585244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.585235    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b248468-49b4-4f41-a435-667efec10460" containerName="payload-pre-processing"
Oct 30 22:35:42.585316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.585308    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="0715e293-39d3-4b44-8429-53ab9827126e" containerName="payload-processing"
Oct 30 22:35:42.585401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.585392    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ebb9a4-b0ac-438b-a5ee-c22b618658dd" containerName="payload-pre-processing"
Oct 30 22:35:42.585478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.585469    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2194013-886a-48da-992e-d00192443e1f" containerName="payload-pre-processing"
Oct 30 22:35:42.607123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.607089    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.694652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.694609    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"]
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.706993    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707060    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707084    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707259    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csb72\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707290    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707418    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707486    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707518    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.709715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.707560    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.763071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.724155    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:42.763622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.733633    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d6b55f-openshift-default-dockercfg-54lpd\""
Oct 30 22:35:42.808495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808457    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.808732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808532    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.808732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808568    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.808732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808618    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.808954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808748    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.808954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808792    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.808954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.808815    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809002    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csb72\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809034    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809376    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809463    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809494    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809662    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.809808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.809787    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.811918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.811896    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.812552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.812530    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.869386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.869316    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.869800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.869417    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csb72\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72\") pod \"e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") " pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:42.974707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:42.973628    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:43.092206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:43.092171    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:43.141589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:43.141479    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:43.571543 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:35:43.571509    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91a055c_7859_4bf5_8e64_25d29b12a7af.slice/crio-478cba9c26daf832e88457ddd60dd1cafae24e832072418a0e67b479e558d439 WatchSource:0}: Error finding container 478cba9c26daf832e88457ddd60dd1cafae24e832072418a0e67b479e558d439: Status 404 returned error can't find the container with id 478cba9c26daf832e88457ddd60dd1cafae24e832072418a0e67b479e558d439
Oct 30 22:35:43.574291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:43.574260    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"]
Oct 30 22:35:43.574634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:43.574607    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:35:43.574678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:43.574669    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:35:43.574737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:43.574720    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:35:44.132982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.132841    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" event={"ID":"d91a055c-7859-4bf5-8e64-25d29b12a7af","Type":"ContainerStarted","Data":"6cf6c97880eb6322f3dc2dde625faca51dbf9e062438f8630355dd97b306f46c"}
Oct 30 22:35:44.132982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.132940    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" event={"ID":"d91a055c-7859-4bf5-8e64-25d29b12a7af","Type":"ContainerStarted","Data":"478cba9c26daf832e88457ddd60dd1cafae24e832072418a0e67b479e558d439"}
Oct 30 22:35:44.159209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.159164    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:44.251544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.251495    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podStartSLOduration=2.25148076 podStartE2EDuration="2.25148076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:44.249140572 +0000 UTC m=+2075.098653607" watchObservedRunningTime="2026-10-30 22:35:44.25148076 +0000 UTC m=+2075.100993732"
Oct 30 22:35:44.317478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.317438    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:44.634304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.634259    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:44.860124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:44.860087    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:45.092484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.092447    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:45.099055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.099025    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:45.116079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.116036    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"]
Oct 30 22:35:45.144578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.144544    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:45.145986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.145961    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:35:45.166457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.166419    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721c75-9fmtg"]
Oct 30 22:35:45.293412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.293383    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b254728e-4059-4043-8c56-fb40628ff45a" path="/var/lib/kubelet/pods/b254728e-4059-4043-8c56-fb40628ff45a/volumes"
Oct 30 22:35:45.556087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:45.556043    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.125884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:47.125848    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.338425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:47.338385    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.525769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:47.525725    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:49.854949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:49.854906    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:51.073882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.073848    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:51.272092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.272047    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:51.487465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.487426    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"]
Oct 30 22:35:51.494599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.494567    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:51.547211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.547178    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"]
Oct 30 22:35:51.603585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.590104    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d6b55f\""
Oct 30 22:35:51.629712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.604169    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d6b55f-dockercfg-pf4l5\""
Oct 30 22:35:51.665911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.665870    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:51.699711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.674811    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6h52\" (UniqueName: \"kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52\") pod \"payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:51.807531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.804822    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:51.807531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.804901    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6h52\" (UniqueName: \"kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52\") pod \"payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:51.807531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.806255    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:51.880491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:51.880455    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6h52\" (UniqueName: \"kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52\") pod \"payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:52.110368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:52.110279    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:52.495601 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:35:52.495567    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcf5007_2264_4a1d_b579_9e6fce8f9f75.slice/crio-e6914e8b56826166c6f0d39588a2b39d883635174a9748ec9ab4481a0e7cf018 WatchSource:0}: Error finding container e6914e8b56826166c6f0d39588a2b39d883635174a9748ec9ab4481a0e7cf018: Status 404 returned error can't find the container with id e6914e8b56826166c6f0d39588a2b39d883635174a9748ec9ab4481a0e7cf018
Oct 30 22:35:52.496921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:52.496897    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"]
Oct 30 22:35:52.631348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:52.631310    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:52.779008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:52.778929    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:53.005131 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.005097    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:53.008517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.008493    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"]
Oct 30 22:35:53.008724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.008679    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy" containerID="cri-o://b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe" gracePeriod=30
Oct 30 22:35:53.296009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.295980    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:35:53.296401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.296030    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" event={"ID":"adcf5007-2264-4a1d-b579-9e6fce8f9f75","Type":"ContainerStarted","Data":"32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427"}
Oct 30 22:35:53.296401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.296053    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" event={"ID":"adcf5007-2264-4a1d-b579-9e6fce8f9f75","Type":"ContainerStarted","Data":"e6914e8b56826166c6f0d39588a2b39d883635174a9748ec9ab4481a0e7cf018"}
Oct 30 22:35:53.411130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:53.411070    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" podStartSLOduration=2.411056139 podStartE2EDuration="2.411056139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:53.4056689 +0000 UTC m=+2084.255181868" watchObservedRunningTime="2026-10-30 22:35:53.411056139 +0000 UTC m=+2084.260569111"
Oct 30 22:35:54.847949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:54.847899    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:55.379467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:55.379430    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:56.226707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:56.224233    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:56.471024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:56.470988    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:58.316975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.316945    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:35:58.344320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.344283    5028 generic.go:356] "Generic (PLEG): container finished" podID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerID="b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe" exitCode=0
Oct 30 22:35:58.344503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.344343    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"
Oct 30 22:35:58.344503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.344366    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" event={"ID":"1a3f5d9c-da8a-48b1-a048-17aa76541c4d","Type":"ContainerDied","Data":"b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe"}
Oct 30 22:35:58.344503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.344435    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl" event={"ID":"1a3f5d9c-da8a-48b1-a048-17aa76541c4d","Type":"ContainerDied","Data":"e4196a1e32e099449a5808adbfb952245800ab847d10f02635a4f4d4abd7f2e9"}
Oct 30 22:35:58.344503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.344467    5028 scope.go:122] "RemoveContainer" containerID="b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe"
Oct 30 22:35:58.361394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.361367    5028 scope.go:122] "RemoveContainer" containerID="c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"
Oct 30 22:35:58.373789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.373765    5028 scope.go:122] "RemoveContainer" containerID="b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe"
Oct 30 22:35:58.374131 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:35:58.374108    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe\": container with ID starting with b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe not found: ID does not exist" containerID="b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe"
Oct 30 22:35:58.374240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.374142    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe"} err="failed to get container status \"b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe\": rpc error: code = NotFound desc = could not find container \"b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe\": container with ID starting with b2dd05bc865c5ff6a864e1cf5170909fd1e99a8de610efe0ae86231c6e28affe not found: ID does not exist"
Oct 30 22:35:58.374240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.374167    5028 scope.go:122] "RemoveContainer" containerID="c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"
Oct 30 22:35:58.374467 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:35:58.374442    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee\": container with ID starting with c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee not found: ID does not exist" containerID="c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"
Oct 30 22:35:58.374507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.374477    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee"} err="failed to get container status \"c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee\": rpc error: code = NotFound desc = could not find container \"c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee\": container with ID starting with c007ab864e44e5a0f5a347267e05de4249c6302181ac2e9d35a2620cf59de9ee not found: ID does not exist"
Oct 30 22:35:58.438426 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:35:58.438385    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:35:58.461611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461575    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.461810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461664    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.461810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461766    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.461810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461794    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.462012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461826    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.462012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461886    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.462012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.461986    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.462174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462056    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.462174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462138    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:58.462174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462162    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert\") pod \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\" (UID: \"1a3f5d9c-da8a-48b1-a048-17aa76541c4d\") "
Oct 30 22:35:58.462378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462251    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:58.462437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462373    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:58.462437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462400    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:58.463260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.462744    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:58.463260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.463176    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.463260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.463197    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.463260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.463211    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.463260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.463224    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.463260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.463238    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.464469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.464443    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:58.464589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.464474    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "kube-api-access-dkrpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:58.464589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.464497    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:58.464905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.464889    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy" pod "1a3f5d9c-da8a-48b1-a048-17aa76541c4d" (UID: "1a3f5d9c-da8a-48b1-a048-17aa76541c4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:58.564816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.564714    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrpf\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-kube-api-access-dkrpf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.564816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.564756    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.564816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.564772    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.564816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.564785    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3f5d9c-da8a-48b1-a048-17aa76541c4d-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:58.765613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.765578    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"]
Oct 30 22:35:58.770075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:58.770051    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-721c75-openshift-default-74d994d999-g9nsl"]
Oct 30 22:35:59.227931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:59.227900    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:59.295581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:35:59.295545    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" path="/var/lib/kubelet/pods/1a3f5d9c-da8a-48b1-a048-17aa76541c4d/volumes"
Oct 30 22:36:00.508284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:00.508246    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:00.858629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:00.858520    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.154187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:01.154097    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.447767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:01.447304    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"]
Oct 30 22:36:01.447767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:01.447372    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-71c235-hsp7d"]
Oct 30 22:36:01.748486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:01.748451    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.907448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:01.907404    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:02.102332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:02.102032    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:02.658015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:02.657964    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:02.869180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:02.869125    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:03.103411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.103375    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"]
Oct 30 22:36:03.150051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.150007    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c3c2cf-j7wt9"]
Oct 30 22:36:03.294305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.294271    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d7248f-9dbe-4255-9ae3-e99ec867256f" path="/var/lib/kubelet/pods/46d7248f-9dbe-4255-9ae3-e99ec867256f/volumes"
Oct 30 22:36:03.294854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.294834    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ced388-6a0b-48f9-8924-7e9aae2ce03c" path="/var/lib/kubelet/pods/b6ced388-6a0b-48f9-8924-7e9aae2ce03c/volumes"
Oct 30 22:36:03.527937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.527889    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:03.707879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.707813    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:03.878102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:03.877975    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:04.472559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:04.472515    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:04.787521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:04.787406    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:04.991599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:04.991565    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"]
Oct 30 22:36:05.037424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:05.037388    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-73735c-fcgwn"]
Oct 30 22:36:05.293153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:05.293121    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73b7ad3-38a8-4b2d-a3be-447c13fffc94" path="/var/lib/kubelet/pods/b73b7ad3-38a8-4b2d-a3be-447c13fffc94/volumes"
Oct 30 22:36:05.313791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:05.313763    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:36:05.455013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:05.454975    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:06.315551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:06.315510    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:06.853514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:06.853473    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:07.010968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.010933    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:07.171497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.171395    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:07.359660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.359624    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"]
Oct 30 22:36:07.360586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.360560    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy"
Oct 30 22:36:07.360586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.360584    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy"
Oct 30 22:36:07.360901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.360882    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy"
Oct 30 22:36:07.366678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.366652    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.420289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.420253    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"]
Oct 30 22:36:07.546767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.543599    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fb3406\""
Oct 30 22:36:07.546767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.543807    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fb3406-dockercfg-j85sg\""
Oct 30 22:36:07.550420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.550381    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.653288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.653242    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4v4\" (UniqueName: \"kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4\") pod \"payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.696198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.696165    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"]
Oct 30 22:36:07.697616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.697587    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy"
Oct 30 22:36:07.697814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.697801    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy"
Oct 30 22:36:07.698161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.698146    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3f5d9c-da8a-48b1-a048-17aa76541c4d" containerName="istio-proxy"
Oct 30 22:36:07.704712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.704669    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"]
Oct 30 22:36:07.704934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.704793    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.806811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.806716    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxckv\" (UniqueName: \"kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv\") pod \"payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.806997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.806830    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4v4\" (UniqueName: \"kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4\") pod \"payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.806997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.806898    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.807120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.807041    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.807785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.807757    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.866350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.866309    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4v4\" (UniqueName: \"kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4\") pod \"payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:07.907842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.907799    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxckv\" (UniqueName: \"kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv\") pod \"payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.908040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.907937    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.908645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.908621    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.967051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.967013    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxckv\" (UniqueName: \"kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv\") pod \"payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:07.985287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:07.985256    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:08.027993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.027955    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:08.377497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.377461    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"]
Oct 30 22:36:08.377866 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:36:08.377476    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6059c1_b900_4cb8_9955_9aa6fc40bb5e.slice/crio-2cc09b19fc6d19e519f5985396ce15775241995a74b969aa65bedc147075a052 WatchSource:0}: Error finding container 2cc09b19fc6d19e519f5985396ce15775241995a74b969aa65bedc147075a052: Status 404 returned error can't find the container with id 2cc09b19fc6d19e519f5985396ce15775241995a74b969aa65bedc147075a052
Oct 30 22:36:08.439661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.439635    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"]
Oct 30 22:36:08.444375 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:36:08.444345    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbb8c2d_a984_48eb_8788_b54a17296b37.slice/crio-068d1d4f14f3c4505ee034e1b633a57f0090a3d8dc1f8bffb3ec003ac10fd4a1 WatchSource:0}: Error finding container 068d1d4f14f3c4505ee034e1b633a57f0090a3d8dc1f8bffb3ec003ac10fd4a1: Status 404 returned error can't find the container with id 068d1d4f14f3c4505ee034e1b633a57f0090a3d8dc1f8bffb3ec003ac10fd4a1
Oct 30 22:36:08.699813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.699716    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" event={"ID":"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e","Type":"ContainerStarted","Data":"c5675a93a2ebbd1f53ee416611e09a5b08d25439e70bf79dbdd17d1fc4481ffc"}
Oct 30 22:36:08.699813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.699782    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" event={"ID":"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e","Type":"ContainerStarted","Data":"2cc09b19fc6d19e519f5985396ce15775241995a74b969aa65bedc147075a052"}
Oct 30 22:36:08.699813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.699812    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:08.701539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.701511    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" event={"ID":"bbbb8c2d-a984-48eb-8788-b54a17296b37","Type":"ContainerStarted","Data":"9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201"}
Oct 30 22:36:08.701675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.701561    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" event={"ID":"bbbb8c2d-a984-48eb-8788-b54a17296b37","Type":"ContainerStarted","Data":"068d1d4f14f3c4505ee034e1b633a57f0090a3d8dc1f8bffb3ec003ac10fd4a1"}
Oct 30 22:36:08.701675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.701593    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:08.828142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.828085    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" podStartSLOduration=1.828069618 podStartE2EDuration="1.828069618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:08.825048984 +0000 UTC m=+2099.674561958" watchObservedRunningTime="2026-10-30 22:36:08.828069618 +0000 UTC m=+2099.677582582"
Oct 30 22:36:08.921700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.921666    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:08.943278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:08.943224    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" podStartSLOduration=1.9432097270000002 podStartE2EDuration="1.943209727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:08.939108524 +0000 UTC m=+2099.788621498" watchObservedRunningTime="2026-10-30 22:36:08.943209727 +0000 UTC m=+2099.792722700"
Oct 30 22:36:09.104748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:09.101719    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:09.262710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:09.262299    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:09.481459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:09.478146    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"]
Oct 30 22:36:09.481459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:09.478399    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" containerID="cri-o://d92af2841a535b666afbaa6a881fa557c8534f0cf4b20f886b48fd0c45e1540d" gracePeriod=30
Oct 30 22:36:09.487523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:09.487494    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:10.240806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.240777    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:36:10.270190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.270155    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"]
Oct 30 22:36:10.275870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.275847    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.276581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.276561    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:36:10.329483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.329453    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"]
Oct 30 22:36:10.364765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.363592    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcrt\" (UniqueName: \"kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.364765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.363666    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.364765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.364002    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.369973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.369942    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:36:10.369973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.369942    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:36:10.419467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.419435    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:36:10.419467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.419449    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:36:10.465666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.465631    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcrt\" (UniqueName: \"kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.465841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.465700    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.465946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.465915    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.466529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.466507    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.468208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.468186    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.524877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.524784    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcrt\" (UniqueName: \"kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.591800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.591769    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:10.853398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.853277    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:10.969739 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:36:10.969702    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcba9495_73fb_4aa4_b7f7_e1411ce03b90.slice/crio-97e7be1a25ef008abefe409aad2abd2116f0032a97649d818e323d6895b77799 WatchSource:0}: Error finding container 97e7be1a25ef008abefe409aad2abd2116f0032a97649d818e323d6895b77799: Status 404 returned error can't find the container with id 97e7be1a25ef008abefe409aad2abd2116f0032a97649d818e323d6895b77799
Oct 30 22:36:10.970505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:10.970444    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"]
Oct 30 22:36:11.012704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.012666    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.183946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.183856    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"]
Oct 30 22:36:11.184124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.184075    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy" containerID="cri-o://4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356" gracePeriod=30
Oct 30 22:36:11.219262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.219228    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.734503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.734469    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:11.753682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.753650    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh" event={"ID":"dcba9495-73fb-4aa4-b7f7-e1411ce03b90","Type":"ContainerStarted","Data":"d9570317dad0c7ad5b0c25cdd95ab15e9cc41d73753d6e2fed60fb7f2bf4bd4d"}
Oct 30 22:36:11.753860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.753723    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh" event={"ID":"dcba9495-73fb-4aa4-b7f7-e1411ce03b90","Type":"ContainerStarted","Data":"97e7be1a25ef008abefe409aad2abd2116f0032a97649d818e323d6895b77799"}
Oct 30 22:36:11.868262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:11.868133    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh" podStartSLOduration=1.8681133330000002 podStartE2EDuration="1.868113333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:11.86464956 +0000 UTC m=+2102.714162534" watchObservedRunningTime="2026-10-30 22:36:11.868113333 +0000 UTC m=+2102.717626308"
Oct 30 22:36:12.537704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.534454    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:12.698245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.698212    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:12.765750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.765711    5028 generic.go:356] "Generic (PLEG): container finished" podID="dcba9495-73fb-4aa4-b7f7-e1411ce03b90" containerID="d9570317dad0c7ad5b0c25cdd95ab15e9cc41d73753d6e2fed60fb7f2bf4bd4d" exitCode=0
Oct 30 22:36:12.766182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.765801    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh" event={"ID":"dcba9495-73fb-4aa4-b7f7-e1411ce03b90","Type":"ContainerDied","Data":"d9570317dad0c7ad5b0c25cdd95ab15e9cc41d73753d6e2fed60fb7f2bf4bd4d"}
Oct 30 22:36:12.865303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.865221    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"]
Oct 30 22:36:12.865505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.865480    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy" containerID="cri-o://fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec" gracePeriod=30
Oct 30 22:36:12.881435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:12.881408    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:13.184339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.184252    5028 patch_prober.go:29] interesting pod/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:36:13.184339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.184310    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:13.520784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.520748    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"]
Oct 30 22:36:13.526036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.526013    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.580508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.580472    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"]
Oct 30 22:36:13.640244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.640198    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn4h\" (UniqueName: \"kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.640458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.640418    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.640592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.640551    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.742113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.742081    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn4h\" (UniqueName: \"kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.742307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.742261    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.742362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.742307    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.743061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.743036    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.744765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.744742    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.803605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.803564    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn4h\" (UniqueName: \"kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.848880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.844945    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:13.910158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.910123    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"]
Oct 30 22:36:13.916591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.916567    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:13.955819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.955781    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:13.955967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.955850    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:13.956032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.955999    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tx2s\" (UniqueName: \"kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:13.968473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:13.968448    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"]
Oct 30 22:36:14.002611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.002578    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:14.018249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.012939    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.057428    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt\" (UniqueName: \"kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt\") pod \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") "
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.057583    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca\") pod \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") "
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.057635    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca\") pod \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\" (UID: \"dcba9495-73fb-4aa4-b7f7-e1411ce03b90\") "
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.058082    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tx2s\" (UniqueName: \"kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.058397    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.058427    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.059337    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca" pod "dcba9495-73fb-4aa4-b7f7-e1411ce03b90" (UID: "dcba9495-73fb-4aa4-b7f7-e1411ce03b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:14.063506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.060612    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.104364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.104318    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt" pod "dcba9495-73fb-4aa4-b7f7-e1411ce03b90" (UID: "dcba9495-73fb-4aa4-b7f7-e1411ce03b90"). InnerVolumeSpecName "kube-api-access-9wcrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:14.104936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.104875    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.108542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.108512    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca" pod "dcba9495-73fb-4aa4-b7f7-e1411ce03b90" (UID: "dcba9495-73fb-4aa4-b7f7-e1411ce03b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:14.133236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.133071    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tx2s\" (UniqueName: \"kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.161709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.161433    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.161709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.161482    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.161709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.161498    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wcrt\" (UniqueName: \"kubernetes.io/projected/dcba9495-73fb-4aa4-b7f7-e1411ce03b90-kube-api-access-9wcrt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.311307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.311235    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:14.341320 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:36:14.341285    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f6ceb9_6a37_4846_918c_2937b080e96d.slice/crio-df91c318db5e08dd8c599901ccfba59ab0dc8efdb095c6440eb651df66008e13 WatchSource:0}: Error finding container df91c318db5e08dd8c599901ccfba59ab0dc8efdb095c6440eb651df66008e13: Status 404 returned error can't find the container with id df91c318db5e08dd8c599901ccfba59ab0dc8efdb095c6440eb651df66008e13
Oct 30 22:36:14.343486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.343458    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"]
Oct 30 22:36:14.702888 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:36:14.702855    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71745e9d_248b_4bb4_b218_cc178f0c83a0.slice/crio-ff0498c4ec647cb9aadcaadc95b95980edfbbcdc6c939660b24cb732b40c05c3 WatchSource:0}: Error finding container ff0498c4ec647cb9aadcaadc95b95980edfbbcdc6c939660b24cb732b40c05c3: Status 404 returned error can't find the container with id ff0498c4ec647cb9aadcaadc95b95980edfbbcdc6c939660b24cb732b40c05c3
Oct 30 22:36:14.712026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.711997    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"]
Oct 30 22:36:14.789028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.788873    5028 generic.go:356] "Generic (PLEG): container finished" podID="5246328d-a9d6-4499-8fd0-c05902265768" containerID="d92af2841a535b666afbaa6a881fa557c8534f0cf4b20f886b48fd0c45e1540d" exitCode=0
Oct 30 22:36:14.789314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.789278    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" event={"ID":"5246328d-a9d6-4499-8fd0-c05902265768","Type":"ContainerDied","Data":"d92af2841a535b666afbaa6a881fa557c8534f0cf4b20f886b48fd0c45e1540d"}
Oct 30 22:36:14.789413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.789373    5028 scope.go:122] "RemoveContainer" containerID="5db3033cf2e5ba970a552d9b29365013c1d6c32e208b523243dc31b6a3cfaedd"
Oct 30 22:36:14.791581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.791553    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m" event={"ID":"71745e9d-248b-4bb4-b218-cc178f0c83a0","Type":"ContainerStarted","Data":"ff0498c4ec647cb9aadcaadc95b95980edfbbcdc6c939660b24cb732b40c05c3"}
Oct 30 22:36:14.793365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.793345    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z" event={"ID":"08f6ceb9-6a37-4846-918c-2937b080e96d","Type":"ContainerStarted","Data":"df91c318db5e08dd8c599901ccfba59ab0dc8efdb095c6440eb651df66008e13"}
Oct 30 22:36:14.794968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.794942    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh" event={"ID":"dcba9495-73fb-4aa4-b7f7-e1411ce03b90","Type":"ContainerDied","Data":"97e7be1a25ef008abefe409aad2abd2116f0032a97649d818e323d6895b77799"}
Oct 30 22:36:14.795178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.795160    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e7be1a25ef008abefe409aad2abd2116f0032a97649d818e323d6895b77799"
Oct 30 22:36:14.795428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.794946    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"
Oct 30 22:36:14.839788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.839768    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:36:14.874424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874335    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874470    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874493    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874546    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874573    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874631    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874705    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874736    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.874855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874765    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs\") pod \"5246328d-a9d6-4499-8fd0-c05902265768\" (UID: \"5246328d-a9d6-4499-8fd0-c05902265768\") "
Oct 30 22:36:14.875083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874858    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:14.875083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.874923    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:14.875083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875045    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:14.875267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875102    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:14.875422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875399    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:14.875749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875728    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5246328d-a9d6-4499-8fd0-c05902265768-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.875798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875757    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.875798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875772    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.875798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875787    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.875897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.875805    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.877343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.877316    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:14.877445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.877337    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "kube-api-access-wctq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:14.877445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.877424    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:14.877556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.877541    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy" pod "5246328d-a9d6-4499-8fd0-c05902265768" (UID: "5246328d-a9d6-4499-8fd0-c05902265768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:14.977147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.977115    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5246328d-a9d6-4499-8fd0-c05902265768-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.977147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.977143    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.977147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.977153    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wctq6\" (UniqueName: \"kubernetes.io/projected/5246328d-a9d6-4499-8fd0-c05902265768-kube-api-access-wctq6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:14.977456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:14.977163    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5246328d-a9d6-4499-8fd0-c05902265768-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:15.805458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:15.805420    5028 generic.go:356] "Generic (PLEG): container finished" podID="08f6ceb9-6a37-4846-918c-2937b080e96d" containerID="4386c5826ef033f74872f7147c0af08d87a01f50f2190ec3fe90f32a54fdedf9" exitCode=0
Oct 30 22:36:15.805672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:15.805514    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z" event={"ID":"08f6ceb9-6a37-4846-918c-2937b080e96d","Type":"ContainerDied","Data":"4386c5826ef033f74872f7147c0af08d87a01f50f2190ec3fe90f32a54fdedf9"}
Oct 30 22:36:15.807106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:15.807086    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"
Oct 30 22:36:15.807207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:15.807138    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs" event={"ID":"5246328d-a9d6-4499-8fd0-c05902265768","Type":"ContainerDied","Data":"640268a8463104a1743d5fe98dbdceb33ea33d44e921695ed6763be65097d2b6"}
Oct 30 22:36:15.807207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:15.807182    5028 scope.go:122] "RemoveContainer" containerID="d92af2841a535b666afbaa6a881fa557c8534f0cf4b20f886b48fd0c45e1540d"
Oct 30 22:36:15.809076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:15.809056    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m" event={"ID":"71745e9d-248b-4bb4-b218-cc178f0c83a0","Type":"ContainerStarted","Data":"82952cc884f8883c441f4a2989f60b78e864e1c72f4b80603bdd2f6413737375"}
Oct 30 22:36:16.029509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.029454    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m" podStartSLOduration=3.029440941 podStartE2EDuration="3.029440941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:16.027268356 +0000 UTC m=+2106.876781333" watchObservedRunningTime="2026-10-30 22:36:16.029440941 +0000 UTC m=+2106.878953954"
Oct 30 22:36:16.208864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.208835    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"]
Oct 30 22:36:16.215164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.215138    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-71c235-openshift-default-5668fb6485-4rszs"]
Oct 30 22:36:16.478114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.478081    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:36:16.603046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.602962    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603040    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603062    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603131    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603240    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603313    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.603455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603347    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.603455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603402    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.603715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603663    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603847    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.603969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.603946    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.604039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.604023    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.604119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.604104    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert\") pod \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\" (UID: \"bcbea46f-84fc-4ea1-9c67-e395f6bc0748\") "
Oct 30 22:36:16.605048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605021    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.605168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605053    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.605168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605072    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.605168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605087    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.605473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605450    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:16.605921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605894    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:16.606014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.605932    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:16.606404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.606386    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "kube-api-access-5pdhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:16.606834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.606819    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token" pod "bcbea46f-84fc-4ea1-9c67-e395f6bc0748" (UID: "bcbea46f-84fc-4ea1-9c67-e395f6bc0748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:16.707179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.707149    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.707179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.707176    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.707179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.707185    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.707427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.707194    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdhp\" (UniqueName: \"kubernetes.io/projected/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-kube-api-access-5pdhp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.707427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.707203    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbea46f-84fc-4ea1-9c67-e395f6bc0748-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:16.819699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.819646    5028 generic.go:356] "Generic (PLEG): container finished" podID="71745e9d-248b-4bb4-b218-cc178f0c83a0" containerID="82952cc884f8883c441f4a2989f60b78e864e1c72f4b80603bdd2f6413737375" exitCode=0
Oct 30 22:36:16.819900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.819739    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m" event={"ID":"71745e9d-248b-4bb4-b218-cc178f0c83a0","Type":"ContainerDied","Data":"82952cc884f8883c441f4a2989f60b78e864e1c72f4b80603bdd2f6413737375"}
Oct 30 22:36:16.822105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.822079    5028 generic.go:356] "Generic (PLEG): container finished" podID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerID="4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356" exitCode=0
Oct 30 22:36:16.822233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.822129    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"
Oct 30 22:36:16.822233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.822164    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" event={"ID":"bcbea46f-84fc-4ea1-9c67-e395f6bc0748","Type":"ContainerDied","Data":"4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356"}
Oct 30 22:36:16.822331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.822231    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx" event={"ID":"bcbea46f-84fc-4ea1-9c67-e395f6bc0748","Type":"ContainerDied","Data":"9152dfeb3045a76f9929f6aa529f2acba4ba72e28f696ac00827f2e1a5edeb15"}
Oct 30 22:36:16.822331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.822261    5028 scope.go:122] "RemoveContainer" containerID="4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356"
Oct 30 22:36:16.837770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.837746    5028 scope.go:122] "RemoveContainer" containerID="e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"
Oct 30 22:36:16.849935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.849902    5028 scope.go:122] "RemoveContainer" containerID="4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356"
Oct 30 22:36:16.850343 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:16.850313    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356\": container with ID starting with 4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356 not found: ID does not exist" containerID="4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356"
Oct 30 22:36:16.850429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.850353    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356"} err="failed to get container status \"4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356\": rpc error: code = NotFound desc = could not find container \"4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356\": container with ID starting with 4604bc5f5155887189ce6e5254f36bffd4e76a8514adb8ba6476524448ee7356 not found: ID does not exist"
Oct 30 22:36:16.850429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.850372    5028 scope.go:122] "RemoveContainer" containerID="e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"
Oct 30 22:36:16.850727 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:16.850672    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341\": container with ID starting with e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341 not found: ID does not exist" containerID="e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"
Oct 30 22:36:16.850792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:16.850727    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341"} err="failed to get container status \"e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341\": rpc error: code = NotFound desc = could not find container \"e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341\": container with ID starting with e6f12127f7ec7b1707fa1994aeb98b9d6085bcfd68d47c00d323c2acbbcbf341 not found: ID does not exist"
Oct 30 22:36:17.031358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.031330    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:17.044258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.044224    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"]
Oct 30 22:36:17.049551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.049521    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c3c2cf-openshift-default-bbfcd86f5-krchx"]
Oct 30 22:36:17.113941 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.113911    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca\") pod \"08f6ceb9-6a37-4846-918c-2937b080e96d\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") "
Oct 30 22:36:17.114134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.113956    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca\") pod \"08f6ceb9-6a37-4846-918c-2937b080e96d\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") "
Oct 30 22:36:17.114190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.114179    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h\" (UniqueName: \"kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h\") pod \"08f6ceb9-6a37-4846-918c-2937b080e96d\" (UID: \"08f6ceb9-6a37-4846-918c-2937b080e96d\") "
Oct 30 22:36:17.114358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.114323    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca" pod "08f6ceb9-6a37-4846-918c-2937b080e96d" (UID: "08f6ceb9-6a37-4846-918c-2937b080e96d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:17.114866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.114842    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:17.116221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.116194    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca" pod "08f6ceb9-6a37-4846-918c-2937b080e96d" (UID: "08f6ceb9-6a37-4846-918c-2937b080e96d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:17.116387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.116365    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h" pod "08f6ceb9-6a37-4846-918c-2937b080e96d" (UID: "08f6ceb9-6a37-4846-918c-2937b080e96d"). InnerVolumeSpecName "kube-api-access-gvn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:17.222377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.222341    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f6ceb9-6a37-4846-918c-2937b080e96d-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:17.222377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.222376    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvn4h\" (UniqueName: \"kubernetes.io/projected/08f6ceb9-6a37-4846-918c-2937b080e96d-kube-api-access-gvn4h\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:17.293019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.292985    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5246328d-a9d6-4499-8fd0-c05902265768" path="/var/lib/kubelet/pods/5246328d-a9d6-4499-8fd0-c05902265768/volumes"
Oct 30 22:36:17.293563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.293549    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" path="/var/lib/kubelet/pods/bcbea46f-84fc-4ea1-9c67-e395f6bc0748/volumes"
Oct 30 22:36:17.836834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.836746    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"
Oct 30 22:36:17.836834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.836755    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z" event={"ID":"08f6ceb9-6a37-4846-918c-2937b080e96d","Type":"ContainerDied","Data":"df91c318db5e08dd8c599901ccfba59ab0dc8efdb095c6440eb651df66008e13"}
Oct 30 22:36:17.836834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:17.836803    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df91c318db5e08dd8c599901ccfba59ab0dc8efdb095c6440eb651df66008e13"
Oct 30 22:36:18.021441 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.021410    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:18.137907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.137878    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s\" (UniqueName: \"kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s\") pod \"71745e9d-248b-4bb4-b218-cc178f0c83a0\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") "
Oct 30 22:36:18.138274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.137948    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca\") pod \"71745e9d-248b-4bb4-b218-cc178f0c83a0\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") "
Oct 30 22:36:18.138274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.138100    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca\") pod \"71745e9d-248b-4bb4-b218-cc178f0c83a0\" (UID: \"71745e9d-248b-4bb4-b218-cc178f0c83a0\") "
Oct 30 22:36:18.138410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.138347    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca" pod "71745e9d-248b-4bb4-b218-cc178f0c83a0" (UID: "71745e9d-248b-4bb4-b218-cc178f0c83a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.138893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.138830    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.140598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.140567    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s" pod "71745e9d-248b-4bb4-b218-cc178f0c83a0" (UID: "71745e9d-248b-4bb4-b218-cc178f0c83a0"). InnerVolumeSpecName "kube-api-access-4tx2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.141242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.141221    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca" pod "71745e9d-248b-4bb4-b218-cc178f0c83a0" (UID: "71745e9d-248b-4bb4-b218-cc178f0c83a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:18.172120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.172093    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:36:18.240072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240033    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240097    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240161    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240247    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240321    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240368    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.240462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240420    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240494    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240524    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.240626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240591    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo\") pod \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\" (UID: \"6ed80fe5-2bbd-4988-a7d9-da7026e965c0\") "
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240518    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240633    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.240869    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241042    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241412    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241439    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71745e9d-248b-4bb4-b218-cc178f0c83a0-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241456    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241471    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241486    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241500    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.241530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.241510    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tx2s\" (UniqueName: \"kubernetes.io/projected/71745e9d-248b-4bb4-b218-cc178f0c83a0-kube-api-access-4tx2s\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.243211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.243176    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.243385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.243289    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.243385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.243325    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "kube-api-access-c5l9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.243508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.243412    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo" pod "6ed80fe5-2bbd-4988-a7d9-da7026e965c0" (UID: "6ed80fe5-2bbd-4988-a7d9-da7026e965c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:18.344113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.343351    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.344113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.343391    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5l9l\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-kube-api-access-c5l9l\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.344113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.343407    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.344113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.343421    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed80fe5-2bbd-4988-a7d9-da7026e965c0-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.847613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.847581    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"
Oct 30 22:36:18.847613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.847592    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m" event={"ID":"71745e9d-248b-4bb4-b218-cc178f0c83a0","Type":"ContainerDied","Data":"ff0498c4ec647cb9aadcaadc95b95980edfbbcdc6c939660b24cb732b40c05c3"}
Oct 30 22:36:18.847922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.847646    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0498c4ec647cb9aadcaadc95b95980edfbbcdc6c939660b24cb732b40c05c3"
Oct 30 22:36:18.849889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.849864    5028 generic.go:356] "Generic (PLEG): container finished" podID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerID="fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec" exitCode=0
Oct 30 22:36:18.850041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.849906    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" event={"ID":"6ed80fe5-2bbd-4988-a7d9-da7026e965c0","Type":"ContainerDied","Data":"fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec"}
Oct 30 22:36:18.850041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.849923    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"
Oct 30 22:36:18.850041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.849948    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j" event={"ID":"6ed80fe5-2bbd-4988-a7d9-da7026e965c0","Type":"ContainerDied","Data":"126e35175c55030f0dd48201e148a0dca9e2a8277e729c678a7f2f78ab3a13ee"}
Oct 30 22:36:18.850041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.849970    5028 scope.go:122] "RemoveContainer" containerID="fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec"
Oct 30 22:36:18.862658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.862631    5028 scope.go:122] "RemoveContainer" containerID="99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"
Oct 30 22:36:18.873666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.873647    5028 scope.go:122] "RemoveContainer" containerID="fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec"
Oct 30 22:36:18.874009 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:18.873989    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec\": container with ID starting with fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec not found: ID does not exist" containerID="fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec"
Oct 30 22:36:18.874060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.874019    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec"} err="failed to get container status \"fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec\": rpc error: code = NotFound desc = could not find container \"fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec\": container with ID starting with fa650f0adc4dac0967a5b4f1b2be15846c7a57b58bf85f22542bf11b78e5beec not found: ID does not exist"
Oct 30 22:36:18.874060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.874036    5028 scope.go:122] "RemoveContainer" containerID="99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"
Oct 30 22:36:18.874335 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:18.874314    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5\": container with ID starting with 99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5 not found: ID does not exist" containerID="99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"
Oct 30 22:36:18.874405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:18.874339    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5"} err="failed to get container status \"99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5\": rpc error: code = NotFound desc = could not find container \"99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5\": container with ID starting with 99085d845e52f1b9ca955768286424a4f3fc68570a554532a33ce2f8b2514ce5 not found: ID does not exist"
Oct 30 22:36:19.012145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:19.012099    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"]
Oct 30 22:36:19.016540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:19.016515    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-73735c-openshift-default-f749cb549-gsq9j"]
Oct 30 22:36:19.078060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:19.078020    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:19.294390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:19.294358    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" path="/var/lib/kubelet/pods/6ed80fe5-2bbd-4988-a7d9-da7026e965c0/volumes"
Oct 30 22:36:19.737245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:19.737216    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:36:19.755862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:19.755833    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:36:21.808321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:21.808286    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"]
Oct 30 22:36:22.108308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:22.108230    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:22.259199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:22.259168    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:23.799052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:23.799017    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:24.052878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:24.052796    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"]
Oct 30 22:36:24.156176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:24.156139    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"]
Oct 30 22:36:24.415057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:24.414978    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:24.580271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:24.580228    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:25.869989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:25.869952    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"]
Oct 30 22:36:25.870448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:25.870160    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy" containerID="cri-o://9e80627354d0378691fb1a78ea368bd75a85621f6c9feec30762bf4c0406b98e" gracePeriod=30
Oct 30 22:36:26.507294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.507258    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"]
Oct 30 22:36:26.507477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.507456    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" containerID="cri-o://9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9" gracePeriod=30
Oct 30 22:36:26.698807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.698770    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:36:26.698988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.698837    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:26.752488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.752456    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"]
Oct 30 22:36:26.752712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.752661    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" containerID="cri-o://0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f" gracePeriod=30
Oct 30 22:36:26.809946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.809755    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg" podUID="1b1c3a7b-f659-4491-976b-255f4448f14e" containerName="payload-pre-processing" containerID="cri-o://6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240" gracePeriod=25
Oct 30 22:36:26.874924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:26.874892    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:27.042051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.041980    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:27.241441 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.241411    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:36:27.335881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.335847    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555\" (UniqueName: \"kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555\") pod \"1b1c3a7b-f659-4491-976b-255f4448f14e\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") "
Oct 30 22:36:27.336064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.335943    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume\") pod \"1b1c3a7b-f659-4491-976b-255f4448f14e\" (UID: \"1b1c3a7b-f659-4491-976b-255f4448f14e\") "
Oct 30 22:36:27.336797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.336398    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume" pod "1b1c3a7b-f659-4491-976b-255f4448f14e" (UID: "1b1c3a7b-f659-4491-976b-255f4448f14e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:27.336797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.336764    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1c3a7b-f659-4491-976b-255f4448f14e-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:27.339234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.339203    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555" pod "1b1c3a7b-f659-4491-976b-255f4448f14e" (UID: "1b1c3a7b-f659-4491-976b-255f4448f14e"). InnerVolumeSpecName "kube-api-access-75555". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:27.438230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:27.438140    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75555\" (UniqueName: \"kubernetes.io/projected/1b1c3a7b-f659-4491-976b-255f4448f14e-kube-api-access-75555\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:28.082732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.079977    5028 generic.go:356] "Generic (PLEG): container finished" podID="1b1c3a7b-f659-4491-976b-255f4448f14e" containerID="6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240" exitCode=0
Oct 30 22:36:28.082732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.080255    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg" event={"ID":"1b1c3a7b-f659-4491-976b-255f4448f14e","Type":"ContainerDied","Data":"6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240"}
Oct 30 22:36:28.082732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.080339    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg" event={"ID":"1b1c3a7b-f659-4491-976b-255f4448f14e","Type":"ContainerDied","Data":"f37ce1def457d3b43ca99aae10bf15104cff206cad0de0d90f219ef77c8d24f5"}
Oct 30 22:36:28.082732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.080368    5028 scope.go:122] "RemoveContainer" containerID="6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240"
Oct 30 22:36:28.082732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.080536    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"
Oct 30 22:36:28.115838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.115815    5028 scope.go:122] "RemoveContainer" containerID="6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240"
Oct 30 22:36:28.116236 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:28.116213    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240\": container with ID starting with 6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240 not found: ID does not exist" containerID="6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240"
Oct 30 22:36:28.116357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.116246    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240"} err="failed to get container status \"6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240\": rpc error: code = NotFound desc = could not find container \"6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240\": container with ID starting with 6e328af5c6cf0f0fb4565afd65151a7a6079772ee2fd3050107398dd24495240 not found: ID does not exist"
Oct 30 22:36:28.267712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.264744    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"]
Oct 30 22:36:28.278713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.278088    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6116bd-6448966987-72btg"]
Oct 30 22:36:28.438259 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:28.438169    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:36:28.777469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:28.777438    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:29.053986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.053887    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld" podUID="0056b572-af70-46a3-8959-e6136b357831" containerName="payload-processing" containerID="cri-o://027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b" gracePeriod=25
Oct 30 22:36:29.157020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.156978    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl" podUID="fc51319e-8440-42eb-a477-b1cff7ac94bf" containerName="payload-pre-processing" containerID="cri-o://3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf" gracePeriod=25
Oct 30 22:36:29.314855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.314776    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1c3a7b-f659-4491-976b-255f4448f14e" path="/var/lib/kubelet/pods/1b1c3a7b-f659-4491-976b-255f4448f14e/volumes"
Oct 30 22:36:29.393455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.393425    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:36:29.439460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.439432    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:36:29.489304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.489269    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume\") pod \"0056b572-af70-46a3-8959-e6136b357831\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") "
Oct 30 22:36:29.489516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.489485    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt\" (UniqueName: \"kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt\") pod \"fc51319e-8440-42eb-a477-b1cff7ac94bf\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") "
Oct 30 22:36:29.489596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.489516    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s\" (UniqueName: \"kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s\") pod \"0056b572-af70-46a3-8959-e6136b357831\" (UID: \"0056b572-af70-46a3-8959-e6136b357831\") "
Oct 30 22:36:29.489666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.489654    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume\") pod \"fc51319e-8440-42eb-a477-b1cff7ac94bf\" (UID: \"fc51319e-8440-42eb-a477-b1cff7ac94bf\") "
Oct 30 22:36:29.489764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.489739    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume" pod "0056b572-af70-46a3-8959-e6136b357831" (UID: "0056b572-af70-46a3-8959-e6136b357831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:29.490082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.490056    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume" pod "fc51319e-8440-42eb-a477-b1cff7ac94bf" (UID: "fc51319e-8440-42eb-a477-b1cff7ac94bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:29.490307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.490272    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51319e-8440-42eb-a477-b1cff7ac94bf-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:29.490307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.490301    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0056b572-af70-46a3-8959-e6136b357831-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:29.492158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.492129    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt" pod "fc51319e-8440-42eb-a477-b1cff7ac94bf" (UID: "fc51319e-8440-42eb-a477-b1cff7ac94bf"). InnerVolumeSpecName "kube-api-access-529bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:29.492403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.492375    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s" pod "0056b572-af70-46a3-8959-e6136b357831" (UID: "0056b572-af70-46a3-8959-e6136b357831"). InnerVolumeSpecName "kube-api-access-wcd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:29.591489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.591404    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529bt\" (UniqueName: \"kubernetes.io/projected/fc51319e-8440-42eb-a477-b1cff7ac94bf-kube-api-access-529bt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:29.591489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:29.591435    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcd9s\" (UniqueName: \"kubernetes.io/projected/0056b572-af70-46a3-8959-e6136b357831-kube-api-access-wcd9s\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:30.100216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.100183    5028 generic.go:356] "Generic (PLEG): container finished" podID="0056b572-af70-46a3-8959-e6136b357831" containerID="027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b" exitCode=0
Oct 30 22:36:30.100413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.100231    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"
Oct 30 22:36:30.100413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.100255    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld" event={"ID":"0056b572-af70-46a3-8959-e6136b357831","Type":"ContainerDied","Data":"027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b"}
Oct 30 22:36:30.100413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.100315    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld" event={"ID":"0056b572-af70-46a3-8959-e6136b357831","Type":"ContainerDied","Data":"dd05b627a3298881c4a61c218a0d72b67e48fdb905490ecf706d1b50708fc3bb"}
Oct 30 22:36:30.100413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.100337    5028 scope.go:122] "RemoveContainer" containerID="027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b"
Oct 30 22:36:30.102076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.102050    5028 generic.go:356] "Generic (PLEG): container finished" podID="fc51319e-8440-42eb-a477-b1cff7ac94bf" containerID="3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf" exitCode=0
Oct 30 22:36:30.102196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.102093    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"
Oct 30 22:36:30.102196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.102148    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl" event={"ID":"fc51319e-8440-42eb-a477-b1cff7ac94bf","Type":"ContainerDied","Data":"3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf"}
Oct 30 22:36:30.102302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.102200    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl" event={"ID":"fc51319e-8440-42eb-a477-b1cff7ac94bf","Type":"ContainerDied","Data":"ef84e6489c4d9b4244d6721dcc621c7cfea2787df1e6ff77c049de07f4159338"}
Oct 30 22:36:30.112300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.112284    5028 scope.go:122] "RemoveContainer" containerID="027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b"
Oct 30 22:36:30.112598 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:30.112572    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b\": container with ID starting with 027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b not found: ID does not exist" containerID="027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b"
Oct 30 22:36:30.112742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.112609    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b"} err="failed to get container status \"027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b\": rpc error: code = NotFound desc = could not find container \"027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b\": container with ID starting with 027844a285857b59c8c38258ad4373b4b17abe8141673f1ba4f441f0f9702e1b not found: ID does not exist"
Oct 30 22:36:30.112742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.112635    5028 scope.go:122] "RemoveContainer" containerID="3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf"
Oct 30 22:36:30.123981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.123842    5028 scope.go:122] "RemoveContainer" containerID="3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf"
Oct 30 22:36:30.124220 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:30.124195    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf\": container with ID starting with 3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf not found: ID does not exist" containerID="3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf"
Oct 30 22:36:30.124295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.124228    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf"} err="failed to get container status \"3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf\": rpc error: code = NotFound desc = could not find container \"3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf\": container with ID starting with 3a5ce8c4c1957d5a130aaea1c5f28b00d6fd763598c8559d569da16dcbcd6ecf not found: ID does not exist"
Oct 30 22:36:30.211942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.211909    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:36:30.212379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.211969    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:30.259800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.259766    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"]
Oct 30 22:36:30.306632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.306600    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7c5d18-57bb7d66f9-lz4ld"]
Oct 30 22:36:30.413631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.413552    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"]
Oct 30 22:36:30.418518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:30.418490    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7c5d18-66dc74c5c5-vcbjl"]
Oct 30 22:36:31.117180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.117140    5028 generic.go:356] "Generic (PLEG): container finished" podID="a1503661-b46b-4acb-98eb-a97d754419e1" containerID="9e80627354d0378691fb1a78ea368bd75a85621f6c9feec30762bf4c0406b98e" exitCode=0
Oct 30 22:36:31.117363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.117259    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" event={"ID":"a1503661-b46b-4acb-98eb-a97d754419e1","Type":"ContainerDied","Data":"9e80627354d0378691fb1a78ea368bd75a85621f6c9feec30762bf4c0406b98e"}
Oct 30 22:36:31.167471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.167446    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:36:31.211963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.211934    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212024    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212094    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212120    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212152    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212228    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212319    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212351    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.212424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212411    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.213216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212429    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket\") pod \"a1503661-b46b-4acb-98eb-a97d754419e1\" (UID: \"a1503661-b46b-4acb-98eb-a97d754419e1\") "
Oct 30 22:36:31.213216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212604    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.213216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212765    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.213216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212856    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.213216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.212969    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:31.213507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.213264    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.213507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.213287    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1503661-b46b-4acb-98eb-a97d754419e1-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.213507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.213302    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.213507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.213316    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.213507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.213330    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.214428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.214402    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:31.214651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.214618    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.214757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.214659    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "kube-api-access-7sbg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:31.214757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.214731    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token" pod "a1503661-b46b-4acb-98eb-a97d754419e1" (UID: "a1503661-b46b-4acb-98eb-a97d754419e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:31.292625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.292591    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0056b572-af70-46a3-8959-e6136b357831" path="/var/lib/kubelet/pods/0056b572-af70-46a3-8959-e6136b357831/volumes"
Oct 30 22:36:31.293035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.293021    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc51319e-8440-42eb-a477-b1cff7ac94bf" path="/var/lib/kubelet/pods/fc51319e-8440-42eb-a477-b1cff7ac94bf/volumes"
Oct 30 22:36:31.315115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.315071    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sbg4\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-kube-api-access-7sbg4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.315296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.315216    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1503661-b46b-4acb-98eb-a97d754419e1-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.315296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.315236    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1503661-b46b-4acb-98eb-a97d754419e1-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.315296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.315251    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1503661-b46b-4acb-98eb-a97d754419e1-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.870098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.870069    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:36:31.924072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.923804    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.924072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.923925    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.924072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.923979    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.924072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.924009    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.924825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.924630    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.924967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.924852    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.924967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.924932    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.924967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.924942    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.925164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925032    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.925227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925202    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.925286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925232    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.925286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925260    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy\") pod \"c864e332-7d36-4997-a38e-e2da5014288e\" (UID: \"c864e332-7d36-4997-a38e-e2da5014288e\") "
Oct 30 22:36:31.925526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925505    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:31.925988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925953    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.925988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925979    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.926148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.925992    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.926148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.926005    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:31.926278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.926171    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:31.926990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.926947    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:31.928787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.928753    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:31.929081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.929042    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "kube-api-access-2g75v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:31.929295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:31.929272    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy" pod "c864e332-7d36-4997-a38e-e2da5014288e" (UID: "c864e332-7d36-4997-a38e-e2da5014288e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:32.027178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.027147    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c864e332-7d36-4997-a38e-e2da5014288e-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.027178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.027178    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g75v\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-kube-api-access-2g75v\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.027431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.027191    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c864e332-7d36-4997-a38e-e2da5014288e-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.027431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.027204    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c864e332-7d36-4997-a38e-e2da5014288e-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.027431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.027216    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c864e332-7d36-4997-a38e-e2da5014288e-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.035143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.035065    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:36:32.127926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.127888    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.128226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.128203    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.128413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.128395    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.128892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.128867    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.129170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.129110    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.129349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.129332    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.129473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.129459    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.129738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.129717    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.129952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.129933    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket\") pod \"720d702d-4c02-44e6-892d-6d55d71092e5\" (UID: \"720d702d-4c02-44e6-892d-6d55d71092e5\") "
Oct 30 22:36:32.130257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.128609    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:32.130334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.130228    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:32.130919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.130895    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.131037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.131023    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.131816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.131788    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:32.131940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.131802    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:32.132104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.132079    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:32.147320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.147281    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:32.168721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.166901    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:32.168721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.167037    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:32.168721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.167135    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68" pod "720d702d-4c02-44e6-892d-6d55d71092e5" (UID: "720d702d-4c02-44e6-892d-6d55d71092e5"). InnerVolumeSpecName "kube-api-access-gpf68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:32.178161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.178114    5028 generic.go:356] "Generic (PLEG): container finished" podID="c864e332-7d36-4997-a38e-e2da5014288e" containerID="9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9" exitCode=0
Oct 30 22:36:32.178338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.178312    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" event={"ID":"c864e332-7d36-4997-a38e-e2da5014288e","Type":"ContainerDied","Data":"9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9"}
Oct 30 22:36:32.178460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.178383    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z" event={"ID":"c864e332-7d36-4997-a38e-e2da5014288e","Type":"ContainerDied","Data":"433bb1d9ad75cdcff3ad7b93cc6f88aadf867c8ad827b87e872de0e2cb921d32"}
Oct 30 22:36:32.178460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.178417    5028 scope.go:122] "RemoveContainer" containerID="9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9"
Oct 30 22:36:32.178805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.178783    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"
Oct 30 22:36:32.182070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.182042    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g" event={"ID":"a1503661-b46b-4acb-98eb-a97d754419e1","Type":"ContainerDied","Data":"7741dc255e569477d8f59a23160e1bf801a7a386c70bd0484badf720e3eb04ed"}
Oct 30 22:36:32.182070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.182062    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"
Oct 30 22:36:32.184405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.184377    5028 generic.go:356] "Generic (PLEG): container finished" podID="720d702d-4c02-44e6-892d-6d55d71092e5" containerID="0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f" exitCode=0
Oct 30 22:36:32.184502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.184407    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"
Oct 30 22:36:32.184502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.184465    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" event={"ID":"720d702d-4c02-44e6-892d-6d55d71092e5","Type":"ContainerDied","Data":"0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f"}
Oct 30 22:36:32.184502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.184498    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9" event={"ID":"720d702d-4c02-44e6-892d-6d55d71092e5","Type":"ContainerDied","Data":"daa0da751014fe5d31062a43311ffb739dcb54b644966dc58c8a7e451ab79d90"}
Oct 30 22:36:32.204776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.204758    5028 scope.go:122] "RemoveContainer" containerID="9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9"
Oct 30 22:36:32.205284 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:32.205262    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9\": container with ID starting with 9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9 not found: ID does not exist" containerID="9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9"
Oct 30 22:36:32.205373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.205290    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9"} err="failed to get container status \"9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9\": rpc error: code = NotFound desc = could not find container \"9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9\": container with ID starting with 9a20f113c284ba583cf1c730ea156c714f6d1e5004c393c20654a7d18c90cfa9 not found: ID does not exist"
Oct 30 22:36:32.205373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.205310    5028 scope.go:122] "RemoveContainer" containerID="9e80627354d0378691fb1a78ea368bd75a85621f6c9feec30762bf4c0406b98e"
Oct 30 22:36:32.222570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.222464    5028 scope.go:122] "RemoveContainer" containerID="0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f"
Oct 30 22:36:32.233170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233139    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.233170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233166    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720d702d-4c02-44e6-892d-6d55d71092e5-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.233361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233177    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.233361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233207    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.233361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233219    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720d702d-4c02-44e6-892d-6d55d71092e5-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.233361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233230    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720d702d-4c02-44e6-892d-6d55d71092e5-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.233361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.233240    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/720d702d-4c02-44e6-892d-6d55d71092e5-kube-api-access-gpf68\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:32.247873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.244813    5028 scope.go:122] "RemoveContainer" containerID="0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f"
Oct 30 22:36:32.247873 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:32.245090    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f\": container with ID starting with 0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f not found: ID does not exist" containerID="0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f"
Oct 30 22:36:32.247873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.245116    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f"} err="failed to get container status \"0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f\": rpc error: code = NotFound desc = could not find container \"0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f\": container with ID starting with 0bf2f80db31583c84c00aa4432b1aea5a507804575f03931af8e018b09d8e39f not found: ID does not exist"
Oct 30 22:36:32.341491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.341342    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"]
Oct 30 22:36:32.346180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.346151    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-rv26g"]
Oct 30 22:36:32.455768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.455730    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"]
Oct 30 22:36:32.458839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.458812    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-q898z"]
Oct 30 22:36:32.563246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.563207    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"]
Oct 30 22:36:32.569863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:32.569833    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-v9dj9"]
Oct 30 22:36:33.292670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:33.292639    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" path="/var/lib/kubelet/pods/720d702d-4c02-44e6-892d-6d55d71092e5/volumes"
Oct 30 22:36:33.293190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:33.293174    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" path="/var/lib/kubelet/pods/a1503661-b46b-4acb-98eb-a97d754419e1/volumes"
Oct 30 22:36:33.293618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:33.293605    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c864e332-7d36-4997-a38e-e2da5014288e" path="/var/lib/kubelet/pods/c864e332-7d36-4997-a38e-e2da5014288e/volumes"
Oct 30 22:36:33.323852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:33.323810    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:37.807238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:37.807201    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:41.145247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:41.145199    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:42.349293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:42.349257    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:46.070502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:46.070464    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:46.219071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:46.219033    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:46.851835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:46.851800    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:50.766344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:50.766297    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.483497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:52.483457    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.756003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:52.755920    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:53.020458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:53.020356    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:53.221926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:53.221892    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"]
Oct 30 22:36:53.265567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:53.265537    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7c5d18-dn25z"]
Oct 30 22:36:53.293744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:53.293639    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f6ceb9-6a37-4846-918c-2937b080e96d" path="/var/lib/kubelet/pods/08f6ceb9-6a37-4846-918c-2937b080e96d/volumes"
Oct 30 22:36:53.666271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:53.666193    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:57.277207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:57.277167    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:58.437291 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:36:58.437247    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:36:59.939026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:36:59.938993    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.099306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:00.099264    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:00.346798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:00.346755    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:01.199093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199054    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"]
Oct 30 22:37:01.199781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199765    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199782    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199797    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199802    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199812    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199819    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199835    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1b1c3a7b-f659-4491-976b-255f4448f14e" containerName="payload-pre-processing"
Oct 30 22:37:01.199844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199840    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1c3a7b-f659-4491-976b-255f4448f14e" containerName="payload-pre-processing"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199849    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199854    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199872    5028 cpu_manager.go:414] "containerMap: removing container" podUID="71745e9d-248b-4bb4-b218-cc178f0c83a0" containerName="revoke-keys"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199877    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71745e9d-248b-4bb4-b218-cc178f0c83a0" containerName="revoke-keys"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199885    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fc51319e-8440-42eb-a477-b1cff7ac94bf" containerName="payload-pre-processing"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199890    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc51319e-8440-42eb-a477-b1cff7ac94bf" containerName="payload-pre-processing"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199897    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199901    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199907    5028 cpu_manager.go:414] "containerMap: removing container" podUID="08f6ceb9-6a37-4846-918c-2937b080e96d" containerName="revoke-keys"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199911    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f6ceb9-6a37-4846-918c-2937b080e96d" containerName="revoke-keys"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199931    5028 cpu_manager.go:414] "containerMap: removing container" podUID="dcba9495-73fb-4aa4-b7f7-e1411ce03b90" containerName="revoke-keys"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199936    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcba9495-73fb-4aa4-b7f7-e1411ce03b90" containerName="revoke-keys"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199945    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199950    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199957    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0056b572-af70-46a3-8959-e6136b357831" containerName="payload-processing"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199961    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0056b572-af70-46a3-8959-e6136b357831" containerName="payload-processing"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199971    5028 cpu_manager.go:414] "containerMap: removing container" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy"
Oct 30 22:37:01.200075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.199976    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200083    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200093    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200098    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200105    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1c3a7b-f659-4491-976b-255f4448f14e" containerName="payload-pre-processing"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200111    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1503661-b46b-4acb-98eb-a97d754419e1" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200120    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="71745e9d-248b-4bb4-b218-cc178f0c83a0" containerName="revoke-keys"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200128    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc51319e-8440-42eb-a477-b1cff7ac94bf" containerName="payload-pre-processing"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200133    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f6ceb9-6a37-4846-918c-2937b080e96d" containerName="revoke-keys"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200140    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200146    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcba9495-73fb-4aa4-b7f7-e1411ce03b90" containerName="revoke-keys"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200152    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c864e332-7d36-4997-a38e-e2da5014288e" containerName="istio-proxy"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200171    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="0056b572-af70-46a3-8959-e6136b357831" containerName="payload-processing"
Oct 30 22:37:01.200622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.200178    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="720d702d-4c02-44e6-892d-6d55d71092e5" containerName="istio-proxy"
Oct 30 22:37:01.205792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.205771    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.262079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.262043    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"]
Oct 30 22:37:01.302517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.302485    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:01.302793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.302544    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:37:01.302793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.302562    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:01.304839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.304817    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:01.341357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.341320    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.341586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.341458    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqsv\" (UniqueName: \"kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.341668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.341604    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.443716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.443654    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.443922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.443778    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqsv\" (UniqueName: \"kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.443922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.443846    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.444386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.444350    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.446493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.446461    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.502821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.502787    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqsv\" (UniqueName: \"kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.522104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.522069    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:01.901739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:01.901709    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"]
Oct 30 22:37:01.901881 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:01.901849    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0fd2ec_a087_4f38_83be_0bb91a8e71d0.slice/crio-96ac06c56bbda5842dfba3a9cfd361b6f066946d3185120c60e7d5c911e44a92 WatchSource:0}: Error finding container 96ac06c56bbda5842dfba3a9cfd361b6f066946d3185120c60e7d5c911e44a92: Status 404 returned error can't find the container with id 96ac06c56bbda5842dfba3a9cfd361b6f066946d3185120c60e7d5c911e44a92
Oct 30 22:37:02.004620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:02.004580    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:02.518347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:02.518241    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr" event={"ID":"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0","Type":"ContainerStarted","Data":"1c4074393f876ebba62e1a86dcd83f1f9c20be9a6e9d7f66870f79878095c38d"}
Oct 30 22:37:02.518347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:02.518316    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr" event={"ID":"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0","Type":"ContainerStarted","Data":"96ac06c56bbda5842dfba3a9cfd361b6f066946d3185120c60e7d5c911e44a92"}
Oct 30 22:37:02.629906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:02.629852    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr" podStartSLOduration=1.629836243 podStartE2EDuration="1.629836243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:02.629110558 +0000 UTC m=+2153.478623532" watchObservedRunningTime="2026-10-30 22:37:02.629836243 +0000 UTC m=+2153.479349262"
Oct 30 22:37:03.532022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:03.531991    5028 generic.go:356] "Generic (PLEG): container finished" podID="8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" containerID="1c4074393f876ebba62e1a86dcd83f1f9c20be9a6e9d7f66870f79878095c38d" exitCode=0
Oct 30 22:37:03.532498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:03.532082    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr" event={"ID":"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0","Type":"ContainerDied","Data":"1c4074393f876ebba62e1a86dcd83f1f9c20be9a6e9d7f66870f79878095c38d"}
Oct 30 22:37:04.729033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.728999    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:04.792730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.792636    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca\") pod \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") "
Oct 30 22:37:04.792902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.792766    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca\") pod \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") "
Oct 30 22:37:04.792955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.792924    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv\" (UniqueName: \"kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv\") pod \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\" (UID: \"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0\") "
Oct 30 22:37:04.793112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.793088    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca" pod "8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" (UID: "8d0fd2ec-a087-4f38-83be-0bb91a8e71d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:04.793379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.793363    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:04.794991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.794959    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv" pod "8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" (UID: "8d0fd2ec-a087-4f38-83be-0bb91a8e71d0"). InnerVolumeSpecName "kube-api-access-vtqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:04.795090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.795005    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca" pod "8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" (UID: "8d0fd2ec-a087-4f38-83be-0bb91a8e71d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:04.895238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.895203    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqsv\" (UniqueName: \"kubernetes.io/projected/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-kube-api-access-vtqsv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:04.895238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:04.895234    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:05.552943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:05.552840    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr" event={"ID":"8d0fd2ec-a087-4f38-83be-0bb91a8e71d0","Type":"ContainerDied","Data":"96ac06c56bbda5842dfba3a9cfd361b6f066946d3185120c60e7d5c911e44a92"}
Oct 30 22:37:05.552943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:05.552877    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"
Oct 30 22:37:05.552943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:05.552905    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ac06c56bbda5842dfba3a9cfd361b6f066946d3185120c60e7d5c911e44a92"
Oct 30 22:37:06.846850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:06.846793    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:07.066451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:07.066415    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:07.218334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:07.218295    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:12.060818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:12.060780    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:12.228767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:12.226553    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:12.378074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:12.377982    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:13.705054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:13.705018    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"]
Oct 30 22:37:13.808273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:13.807553    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"]
Oct 30 22:37:14.060009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:14.058815    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:14.222450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:14.222409    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:16.762559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:16.762516    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:17.511534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:17.511498    5028 scope.go:122] "RemoveContainer" containerID="0e3fa3ec2fba96e08869f4fd0188a597ea2c1fa1181938ca4f3db3d0d791ab75"
Oct 30 22:37:18.706282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:18.706239    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l" podUID="9cc6a178-d6b4-4a68-ac89-12013e30d113" containerName="payload-processing" containerID="cri-o://3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e" gracePeriod=25
Oct 30 22:37:18.808385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:18.808342    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6" podUID="94e9157c-68a6-4bde-a2c2-8782222c725b" containerName="payload-pre-processing" containerID="cri-o://2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5" gracePeriod=25
Oct 30 22:37:19.036390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.035969    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:37:19.113928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.113902    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:37:19.181540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.181508    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47\" (UniqueName: \"kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47\") pod \"9cc6a178-d6b4-4a68-ac89-12013e30d113\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") "
Oct 30 22:37:19.181773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.181624    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume\") pod \"9cc6a178-d6b4-4a68-ac89-12013e30d113\" (UID: \"9cc6a178-d6b4-4a68-ac89-12013e30d113\") "
Oct 30 22:37:19.182331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.182302    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume" pod "9cc6a178-d6b4-4a68-ac89-12013e30d113" (UID: "9cc6a178-d6b4-4a68-ac89-12013e30d113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:19.184332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.184304    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47" pod "9cc6a178-d6b4-4a68-ac89-12013e30d113" (UID: "9cc6a178-d6b4-4a68-ac89-12013e30d113"). InnerVolumeSpecName "kube-api-access-7vx47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:19.283003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.282909    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume\") pod \"94e9157c-68a6-4bde-a2c2-8782222c725b\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") "
Oct 30 22:37:19.283194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.283030    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m\" (UniqueName: \"kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m\") pod \"94e9157c-68a6-4bde-a2c2-8782222c725b\" (UID: \"94e9157c-68a6-4bde-a2c2-8782222c725b\") "
Oct 30 22:37:19.284573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.283498    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume" pod "94e9157c-68a6-4bde-a2c2-8782222c725b" (UID: "94e9157c-68a6-4bde-a2c2-8782222c725b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:19.284573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.283733    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc6a178-d6b4-4a68-ac89-12013e30d113-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:19.284573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.283757    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e9157c-68a6-4bde-a2c2-8782222c725b-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:19.284573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.283770    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vx47\" (UniqueName: \"kubernetes.io/projected/9cc6a178-d6b4-4a68-ac89-12013e30d113-kube-api-access-7vx47\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:19.287088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.287046    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m" pod "94e9157c-68a6-4bde-a2c2-8782222c725b" (UID: "94e9157c-68a6-4bde-a2c2-8782222c725b"). InnerVolumeSpecName "kube-api-access-grz8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:19.385329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.385295    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grz8m\" (UniqueName: \"kubernetes.io/projected/94e9157c-68a6-4bde-a2c2-8782222c725b-kube-api-access-grz8m\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:19.738438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.738410    5028 generic.go:356] "Generic (PLEG): container finished" podID="9cc6a178-d6b4-4a68-ac89-12013e30d113" containerID="3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e" exitCode=0
Oct 30 22:37:19.738894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.738450    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"
Oct 30 22:37:19.738894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.738487    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l" event={"ID":"9cc6a178-d6b4-4a68-ac89-12013e30d113","Type":"ContainerDied","Data":"3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e"}
Oct 30 22:37:19.738894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.738552    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l" event={"ID":"9cc6a178-d6b4-4a68-ac89-12013e30d113","Type":"ContainerDied","Data":"018d172e94d553bde3b14e0f93f47b3b7dbe2b0f912367a65a81996842fdf059"}
Oct 30 22:37:19.738894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.738583    5028 scope.go:122] "RemoveContainer" containerID="3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e"
Oct 30 22:37:19.740677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.740649    5028 generic.go:356] "Generic (PLEG): container finished" podID="94e9157c-68a6-4bde-a2c2-8782222c725b" containerID="2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5" exitCode=0
Oct 30 22:37:19.740841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.740735    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6" event={"ID":"94e9157c-68a6-4bde-a2c2-8782222c725b","Type":"ContainerDied","Data":"2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5"}
Oct 30 22:37:19.740841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.740815    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6" event={"ID":"94e9157c-68a6-4bde-a2c2-8782222c725b","Type":"ContainerDied","Data":"b506a6535a3608c89e3d608b38740d799812deef3c86e73c9aa7ee7084138978"}
Oct 30 22:37:19.741083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.741055    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"
Oct 30 22:37:19.755876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.755849    5028 scope.go:122] "RemoveContainer" containerID="3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e"
Oct 30 22:37:19.756774 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:19.756746    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e\": container with ID starting with 3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e not found: ID does not exist" containerID="3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e"
Oct 30 22:37:19.756876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.756776    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e"} err="failed to get container status \"3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e\": rpc error: code = NotFound desc = could not find container \"3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e\": container with ID starting with 3ee227ce34c934f2220422ec9ebba37693d2b6e2a99a6d6fdf63150d8689ab8e not found: ID does not exist"
Oct 30 22:37:19.756876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.756794    5028 scope.go:122] "RemoveContainer" containerID="2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5"
Oct 30 22:37:19.767582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.767562    5028 scope.go:122] "RemoveContainer" containerID="2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5"
Oct 30 22:37:19.767878 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:19.767856    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5\": container with ID starting with 2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5 not found: ID does not exist" containerID="2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5"
Oct 30 22:37:19.767943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.767887    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5"} err="failed to get container status \"2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5\": rpc error: code = NotFound desc = could not find container \"2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5\": container with ID starting with 2a7e1003b51c7b8ae050dcbdd0ea0dad3d9ed2342ae94180c00c26f2a29742a5 not found: ID does not exist"
Oct 30 22:37:19.899003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.898969    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"]
Oct 30 22:37:19.948934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:19.948897    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-bbd10f-5f8cd8b7f5-4fv8l"]
Oct 30 22:37:20.008392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:20.008302    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"]
Oct 30 22:37:20.012727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:20.012697    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-bbd10f-6f97bbbb8c-v6zl6"]
Oct 30 22:37:21.229277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:21.229210    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:21.304907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:21.304872    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e9157c-68a6-4bde-a2c2-8782222c725b" path="/var/lib/kubelet/pods/94e9157c-68a6-4bde-a2c2-8782222c725b/volumes"
Oct 30 22:37:21.305432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:21.305417    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc6a178-d6b4-4a68-ac89-12013e30d113" path="/var/lib/kubelet/pods/9cc6a178-d6b4-4a68-ac89-12013e30d113/volumes"
Oct 30 22:37:21.382982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:21.382936    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:21.533754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:21.533651    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:23.055674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.055622    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"]
Oct 30 22:37:23.103358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.103319    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e0aaca-dxkcw"]
Oct 30 22:37:23.293259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.293221    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4ac25f-59b3-4334-bef5-ed89ae2de00e" path="/var/lib/kubelet/pods/ad4ac25f-59b3-4334-bef5-ed89ae2de00e/volumes"
Oct 30 22:37:23.755369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.755320    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"]
Oct 30 22:37:23.756907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756876    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy"
Oct 30 22:37:23.756907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756904    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756934    5028 cpu_manager.go:414] "containerMap: removing container" podUID="94e9157c-68a6-4bde-a2c2-8782222c725b" containerName="payload-pre-processing"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756941    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e9157c-68a6-4bde-a2c2-8782222c725b" containerName="payload-pre-processing"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756976    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" containerName="revoke-keys"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756982    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" containerName="revoke-keys"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.756999    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9cc6a178-d6b4-4a68-ac89-12013e30d113" containerName="payload-processing"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757004    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc6a178-d6b4-4a68-ac89-12013e30d113" containerName="payload-processing"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757027    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy"
Oct 30 22:37:23.757124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757031    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbea46f-84fc-4ea1-9c67-e395f6bc0748" containerName="istio-proxy"
Oct 30 22:37:23.757554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757271    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed80fe5-2bbd-4988-a7d9-da7026e965c0" containerName="istio-proxy"
Oct 30 22:37:23.757554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757286    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5246328d-a9d6-4499-8fd0-c05902265768" containerName="istio-proxy"
Oct 30 22:37:23.757554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757300    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e9157c-68a6-4bde-a2c2-8782222c725b" containerName="payload-pre-processing"
Oct 30 22:37:23.757554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757311    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" containerName="revoke-keys"
Oct 30 22:37:23.757554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.757319    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc6a178-d6b4-4a68-ac89-12013e30d113" containerName="payload-processing"
Oct 30 22:37:23.762527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.762505    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.866091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.866053    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"]
Oct 30 22:37:23.952315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952280    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5ps\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952330    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952433    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952467    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952583    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952631    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952713    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952827    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:23.952938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:23.952894    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.054681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.054585    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.054681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.054664    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.054904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.054861    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5ps\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.054963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.054907    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.054963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.054946    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.054981    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055037    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055068    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055133    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055146    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055182    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055399    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.055613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.055583    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.056054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.056028    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.058214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.058181    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.058342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.058183    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.114604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.114574    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.114781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.114614    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5ps\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.378875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.378781    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:24.804677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.804638    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"]
Oct 30 22:37:24.836863 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:24.836827    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d252e95_ab99_45f6_8554_e0296a60de1a.slice/crio-19709f5fc52c8311386096705fca9d927511f833fc1dbc77d3fae45710074ef6 WatchSource:0}: Error finding container 19709f5fc52c8311386096705fca9d927511f833fc1dbc77d3fae45710074ef6: Status 404 returned error can't find the container with id 19709f5fc52c8311386096705fca9d927511f833fc1dbc77d3fae45710074ef6
Oct 30 22:37:24.839743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.839709    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:24.839849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.839780    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:24.839849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:24.839811    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:25.798681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:25.798641    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" event={"ID":"7d252e95-ab99-45f6-8554-e0296a60de1a","Type":"ContainerStarted","Data":"d30b3aec014d739a62f103c61fb6c49007d906613ef29e37256cf77e2ebc26d2"}
Oct 30 22:37:25.799207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:25.798765    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" event={"ID":"7d252e95-ab99-45f6-8554-e0296a60de1a","Type":"ContainerStarted","Data":"19709f5fc52c8311386096705fca9d927511f833fc1dbc77d3fae45710074ef6"}
Oct 30 22:37:25.964506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:25.964437    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podStartSLOduration=2.9644150099999997 podStartE2EDuration="2.96441501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:25.962009583 +0000 UTC m=+2176.811522558" watchObservedRunningTime="2026-10-30 22:37:25.96441501 +0000 UTC m=+2176.813927979"
Oct 30 22:37:26.043134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:26.043101    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:26.379453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:26.379418    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:26.381425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:26.381398    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 22:37:26.381532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:26.381448    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 22:37:27.379738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:27.379667    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 22:37:27.380179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:27.379760    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 22:37:28.379911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:28.379874    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 22:37:28.380346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:28.379928    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 22:37:28.465674 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:28.465628    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:37:29.379911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:29.379878    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 22:37:29.380391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:29.379944    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 22:37:30.082971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:30.082898    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:30.381152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:30.381066    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 22:37:30.381785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:30.381752    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 22:37:31.499823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:31.499787    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:31.785059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:31.784957    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:31.957745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:31.955481    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:31.998620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:31.998589    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:31.999593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:31.999545    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:32.001985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:32.001959    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:37:32.025923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:32.025888    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"]
Oct 30 22:37:32.077081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:32.076985    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-6116bd-7zhbh"]
Oct 30 22:37:32.316022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:32.315989    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:32.467954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:32.467919    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:32.696702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:32.696651    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:33.292474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:33.292437    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcba9495-73fb-4aa4-b7f7-e1411ce03b90" path="/var/lib/kubelet/pods/dcba9495-73fb-4aa4-b7f7-e1411ce03b90/volumes"
Oct 30 22:37:34.798011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:34.797980    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-ae3692"]
Oct 30 22:37:34.802586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:34.802566    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:34.856342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:34.856309    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-ae3692"]
Oct 30 22:37:34.871891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:34.871850    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29f6d\" (UniqueName: \"kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d\") pod \"test-curl-7135-ae3692\" (UID: \"545e50e4-e28d-4dfe-b646-780b3fc534a5\") " pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:34.897232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:34.897199    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9v4z\""
Oct 30 22:37:34.973912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:34.973876    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29f6d\" (UniqueName: \"kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d\") pod \"test-curl-7135-ae3692\" (UID: \"545e50e4-e28d-4dfe-b646-780b3fc534a5\") " pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:35.033060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:35.033028    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29f6d\" (UniqueName: \"kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d\") pod \"test-curl-7135-ae3692\" (UID: \"545e50e4-e28d-4dfe-b646-780b3fc534a5\") " pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:35.116748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:35.116635    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:35.500535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:35.500499    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-ae3692"]
Oct 30 22:37:35.501222 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:35.501195    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545e50e4_e28d_4dfe_b646_780b3fc534a5.slice/crio-245220c4d15f6332c46e99a93eaa775050a0198e13c482312167e807c3e236f4 WatchSource:0}: Error finding container 245220c4d15f6332c46e99a93eaa775050a0198e13c482312167e807c3e236f4: Status 404 returned error can't find the container with id 245220c4d15f6332c46e99a93eaa775050a0198e13c482312167e807c3e236f4
Oct 30 22:37:35.942321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:35.942227    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-ae3692" event={"ID":"545e50e4-e28d-4dfe-b646-780b3fc534a5","Type":"ContainerStarted","Data":"e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c"}
Oct 30 22:37:35.942321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:35.942285    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-ae3692" event={"ID":"545e50e4-e28d-4dfe-b646-780b3fc534a5","Type":"ContainerStarted","Data":"245220c4d15f6332c46e99a93eaa775050a0198e13c482312167e807c3e236f4"}
Oct 30 22:37:36.059748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:36.059668    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-ae3692" podStartSLOduration=2.059647626 podStartE2EDuration="2.059647626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:36.054339435 +0000 UTC m=+2186.903852408" watchObservedRunningTime="2026-10-30 22:37:36.059647626 +0000 UTC m=+2186.909160600"
Oct 30 22:37:36.117892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:36.117853    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:37.297295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:37.297242    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:37.683615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:37.683528    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-ae3692"]
Oct 30 22:37:37.691700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:37.690719    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-ae3692"]
Oct 30 22:37:37.839707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:37.818232    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:37.997144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:37.997106    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-ae3692" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" containerName="test-curl-7135-ae3692" containerID="cri-o://e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c" gracePeriod=2
Oct 30 22:37:38.047383 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:38.047339    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-ae3692\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:38.066151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.066122    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:38.321836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.321732    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"]
Oct 30 22:37:38.327388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.327359    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3600a-qtbsf"]
Oct 30 22:37:38.742386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.742337    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:38.748279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.748251    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"]
Oct 30 22:37:38.748975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.748960    5028 cpu_manager.go:414] "containerMap: removing container" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" containerName="test-curl-7135-ae3692"
Oct 30 22:37:38.749043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.748976    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" containerName="test-curl-7135-ae3692"
Oct 30 22:37:38.749106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.749096    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" containerName="test-curl-7135-ae3692"
Oct 30 22:37:38.753090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.753072    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.804261 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:38.804225    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-ae3692\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:38.812747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.812717    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"]
Oct 30 22:37:38.863081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.863050    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:38.863081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.863081    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:38.865862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.865834    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.865983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.865903    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcz8\" (UniqueName: \"kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.866024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.865991    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.900956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.900923    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:38.901173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.901021    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:37:38.906174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.906148    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"]
Oct 30 22:37:38.910836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.910818    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:38.967204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.967163    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcz8\" (UniqueName: \"kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.967482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.967446    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.967785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.967764    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.968805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.968780    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:38.969377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.969355    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"]
Oct 30 22:37:38.970634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:38.970608    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:39.027946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.027873    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcz8\" (UniqueName: \"kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8\") pod \"maas-api-revoke-keys-e2e-mt-29bbe735-98l5s\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:39.067173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.067141    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:39.068923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.068885    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.068951    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcgs\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069042    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069078    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069154    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069276    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069344    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069474    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.069544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.069516    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.081220 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:39.081185    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-ae3692\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171374    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171426    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcgs\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171461    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171485    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171507    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171646    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171863    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171899    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.171998    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.172105    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.172410    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.172496    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.172679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.172618    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.173617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.172863    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.175136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.175100    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.182984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.180263    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.194187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.194159    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:39.254164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.254133    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.256200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.256158    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcgs\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.307627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.307552    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570f3dab-f32a-46b4-a89c-bec3615181d5" path="/var/lib/kubelet/pods/570f3dab-f32a-46b4-a89c-bec3615181d5/volumes"
Oct 30 22:37:39.345898 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:39.345857    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-ae3692\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:39.353614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.353587    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:39.452194 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:39.452153    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188ff58b_dfa2_4a64_9135_616e23d35641.slice/crio-8808f4af593987430b9fa1055e4c0998d78c0324240704f2c203187fb8571a12 WatchSource:0}: Error finding container 8808f4af593987430b9fa1055e4c0998d78c0324240704f2c203187fb8571a12: Status 404 returned error can't find the container with id 8808f4af593987430b9fa1055e4c0998d78c0324240704f2c203187fb8571a12
Oct 30 22:37:39.452194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.452162    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"]
Oct 30 22:37:39.527837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.527779    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:39.542227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.542204    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:39.779380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.779342    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"]
Oct 30 22:37:39.783408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.783380    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d57980-djj9m"]
Oct 30 22:37:39.983235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.983202    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"]
Oct 30 22:37:39.983235 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:39.983216    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0ed72a_3967_4cc3_8173_4a0c1fccd7f8.slice/crio-3b82e7ad9c147319252260927d69619d72a51a280cb32ed92baf240ca6a5c5d4 WatchSource:0}: Error finding container 3b82e7ad9c147319252260927d69619d72a51a280cb32ed92baf240ca6a5c5d4: Status 404 returned error can't find the container with id 3b82e7ad9c147319252260927d69619d72a51a280cb32ed92baf240ca6a5c5d4
Oct 30 22:37:39.986748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.986710    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:39.986885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.986795    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:39.986885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:39.986834    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:40.021211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.021171    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s" event={"ID":"188ff58b-dfa2-4a64-9135-616e23d35641","Type":"ContainerStarted","Data":"16c458cf99ee5b39e46fb293b2d14ebf3f3965bec7ac97e90342bad54cd8b3ef"}
Oct 30 22:37:40.021211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.021240    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s" event={"ID":"188ff58b-dfa2-4a64-9135-616e23d35641","Type":"ContainerStarted","Data":"8808f4af593987430b9fa1055e4c0998d78c0324240704f2c203187fb8571a12"}
Oct 30 22:37:40.023389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.023354    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" event={"ID":"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8","Type":"ContainerStarted","Data":"3b82e7ad9c147319252260927d69619d72a51a280cb32ed92baf240ca6a5c5d4"}
Oct 30 22:37:40.044656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.044626    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:40.138331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.138267    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s" podStartSLOduration=2.138247915 podStartE2EDuration="2.138247915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:40.134105865 +0000 UTC m=+2190.983618837" watchObservedRunningTime="2026-10-30 22:37:40.138247915 +0000 UTC m=+2190.987760887"
Oct 30 22:37:40.199173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.199135    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:40.265443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.265416    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:40.390902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.390869    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:40.394840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.391676    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d\" (UniqueName: \"kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d\") pod \"545e50e4-e28d-4dfe-b646-780b3fc534a5\" (UID: \"545e50e4-e28d-4dfe-b646-780b3fc534a5\") "
Oct 30 22:37:40.398734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.398365    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d" pod "545e50e4-e28d-4dfe-b646-780b3fc534a5" (UID: "545e50e4-e28d-4dfe-b646-780b3fc534a5"). InnerVolumeSpecName "kube-api-access-29f6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:40.422010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.421979    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"]
Oct 30 22:37:40.422258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.422230    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy" containerID="cri-o://6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0" gracePeriod=30
Oct 30 22:37:40.494718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.494618    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29f6d\" (UniqueName: \"kubernetes.io/projected/545e50e4-e28d-4dfe-b646-780b3fc534a5-kube-api-access-29f6d\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:40.703710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:40.701128    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:41.051355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.051321    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" event={"ID":"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8","Type":"ContainerStarted","Data":"e8391c72614d212e80b08e88442f9bcc5151d9d9194ba7df8929887329e28298"}
Oct 30 22:37:41.052882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.052851    5028 generic.go:356] "Generic (PLEG): container finished" podID="545e50e4-e28d-4dfe-b646-780b3fc534a5" containerID="e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c" exitCode=137
Oct 30 22:37:41.052882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.052882    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:41.053080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.052943    5028 scope.go:122] "RemoveContainer" containerID="e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c"
Oct 30 22:37:41.054809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.054780    5028 generic.go:356] "Generic (PLEG): container finished" podID="188ff58b-dfa2-4a64-9135-616e23d35641" containerID="16c458cf99ee5b39e46fb293b2d14ebf3f3965bec7ac97e90342bad54cd8b3ef" exitCode=0
Oct 30 22:37:41.054919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.054817    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s" event={"ID":"188ff58b-dfa2-4a64-9135-616e23d35641","Type":"ContainerDied","Data":"16c458cf99ee5b39e46fb293b2d14ebf3f3965bec7ac97e90342bad54cd8b3ef"}
Oct 30 22:37:41.064427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.064405    5028 scope.go:122] "RemoveContainer" containerID="e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c"
Oct 30 22:37:41.064731 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:41.064700    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c\": container with ID starting with e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c not found: ID does not exist" containerID="e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c"
Oct 30 22:37:41.064796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.064735    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c"} err="failed to get container status \"e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c\": rpc error: code = NotFound desc = could not find container \"e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c\": container with ID starting with e88709355866a3594a415cc5657f9804a0a7a6ce567c45b809f7b249f94d5f3c not found: ID does not exist"
Oct 30 22:37:41.170168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.170114    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podStartSLOduration=3.170100943 podStartE2EDuration="3.170100943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:41.168495946 +0000 UTC m=+2192.018008919" watchObservedRunningTime="2026-10-30 22:37:41.170100943 +0000 UTC m=+2192.019613914"
Oct 30 22:37:41.230712 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:41.221913    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-ae3692\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" pod="opendatahub/test-curl-7135-ae3692"
Oct 30 22:37:41.292589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.292513    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545e50e4-e28d-4dfe-b646-780b3fc534a5" path="/var/lib/kubelet/pods/545e50e4-e28d-4dfe-b646-780b3fc534a5/volumes"
Oct 30 22:37:41.293063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.293048    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71745e9d-248b-4bb4-b218-cc178f0c83a0" path="/var/lib/kubelet/pods/71745e9d-248b-4bb4-b218-cc178f0c83a0/volumes"
Oct 30 22:37:41.463920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.463887    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:41.542773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.537178    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:41.542773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.541974    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 22:37:41.542773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.542036    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 22:37:41.647780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:41.647648    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:42.304624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.304592    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:42.458195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.458165    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8\" (UniqueName: \"kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8\") pod \"188ff58b-dfa2-4a64-9135-616e23d35641\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") "
Oct 30 22:37:42.458354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.458333    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca\") pod \"188ff58b-dfa2-4a64-9135-616e23d35641\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") "
Oct 30 22:37:42.458476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.458452    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca\") pod \"188ff58b-dfa2-4a64-9135-616e23d35641\" (UID: \"188ff58b-dfa2-4a64-9135-616e23d35641\") "
Oct 30 22:37:42.458914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.458886    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca" pod "188ff58b-dfa2-4a64-9135-616e23d35641" (UID: "188ff58b-dfa2-4a64-9135-616e23d35641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:42.459568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.459542    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:42.460604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.460577    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8" pod "188ff58b-dfa2-4a64-9135-616e23d35641" (UID: "188ff58b-dfa2-4a64-9135-616e23d35641"). InnerVolumeSpecName "kube-api-access-zwcz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:42.461861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.461833    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca" pod "188ff58b-dfa2-4a64-9135-616e23d35641" (UID: "188ff58b-dfa2-4a64-9135-616e23d35641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:42.542711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.542048    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 22:37:42.542711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.542103    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 22:37:42.560895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.560867    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188ff58b-dfa2-4a64-9135-616e23d35641-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:42.560895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:42.560893    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcz8\" (UniqueName: \"kubernetes.io/projected/188ff58b-dfa2-4a64-9135-616e23d35641-kube-api-access-zwcz8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:43.077092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:43.077059    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"
Oct 30 22:37:43.077092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:43.077077    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s" event={"ID":"188ff58b-dfa2-4a64-9135-616e23d35641","Type":"ContainerDied","Data":"8808f4af593987430b9fa1055e4c0998d78c0324240704f2c203187fb8571a12"}
Oct 30 22:37:43.077326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:43.077137    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8808f4af593987430b9fa1055e4c0998d78c0324240704f2c203187fb8571a12"
Oct 30 22:37:43.534712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:43.528387    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 22:37:43.534712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:43.528462    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 22:37:44.197523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:44.197487    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:44.545715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:44.545043    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 22:37:44.545715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:44.545108    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 22:37:45.093408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.093241    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:45.293732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.293696    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:45.462848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.461256    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:45.468672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.468638    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"]
Oct 30 22:37:45.469497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.468937    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" containerID="cri-o://478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31" gracePeriod=30
Oct 30 22:37:45.562710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.559292    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 22:37:45.562710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.559351    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 22:37:45.904333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.904306    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:37:45.980044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980009    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980050    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980116    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980163    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980235    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980316    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980331    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:45.980504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980396    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980494    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.980642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980544    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket\") pod \"74b31fa1-9460-47b4-bddc-080508b9c5b7\" (UID: \"74b31fa1-9460-47b4-bddc-080508b9c5b7\") "
Oct 30 22:37:45.981212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.980981    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:45.981212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.981068    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:45.981212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.981174    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:45.981387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.981243    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:45.981387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.981264    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:45.981387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.981286    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:45.981387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.981301    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b31fa1-9460-47b4-bddc-080508b9c5b7-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:45.983125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.983102    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:45.983207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.983138    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:37:45.983582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.983561    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:45.983663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:45.983641    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj" pod "74b31fa1-9460-47b4-bddc-080508b9c5b7" (UID: "74b31fa1-9460-47b4-bddc-080508b9c5b7"). InnerVolumeSpecName "kube-api-access-kddzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:46.083570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.083529    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.083570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.083566    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.083886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.083581    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.083886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.083595    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.083886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.083609    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kddzj\" (UniqueName: \"kubernetes.io/projected/74b31fa1-9460-47b4-bddc-080508b9c5b7-kube-api-access-kddzj\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.083886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.083624    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b31fa1-9460-47b4-bddc-080508b9c5b7-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.108991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.108956    5028 generic.go:356] "Generic (PLEG): container finished" podID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerID="6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0" exitCode=0
Oct 30 22:37:46.108991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.109002    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"
Oct 30 22:37:46.109284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.109039    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" event={"ID":"74b31fa1-9460-47b4-bddc-080508b9c5b7","Type":"ContainerDied","Data":"6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0"}
Oct 30 22:37:46.109284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.109091    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk" event={"ID":"74b31fa1-9460-47b4-bddc-080508b9c5b7","Type":"ContainerDied","Data":"09374de662e289a7ea2f9a9132d071a079ea3a07915e6cb408f9661a867eb4e5"}
Oct 30 22:37:46.109284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.109115    5028 scope.go:122] "RemoveContainer" containerID="6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0"
Oct 30 22:37:46.153450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.153420    5028 scope.go:122] "RemoveContainer" containerID="bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"
Oct 30 22:37:46.165834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.165813    5028 scope.go:122] "RemoveContainer" containerID="6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0"
Oct 30 22:37:46.166314 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:46.166286    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0\": container with ID starting with 6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0 not found: ID does not exist" containerID="6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0"
Oct 30 22:37:46.166431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.166320    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0"} err="failed to get container status \"6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0\": rpc error: code = NotFound desc = could not find container \"6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0\": container with ID starting with 6a156dc7a7c42a3c8e6f2ca0efff029ce1d72b969bf0b44d7f7ed200295141a0 not found: ID does not exist"
Oct 30 22:37:46.166431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.166345    5028 scope.go:122] "RemoveContainer" containerID="bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"
Oct 30 22:37:46.166635 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:46.166615    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33\": container with ID starting with bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33 not found: ID does not exist" containerID="bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"
Oct 30 22:37:46.166796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.166650    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33"} err="failed to get container status \"bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33\": rpc error: code = NotFound desc = could not find container \"bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33\": container with ID starting with bfa6e7af1db768e974f7f468ffdef99cc4eb0d98051a0a6353a9123b2716dd33 not found: ID does not exist"
Oct 30 22:37:46.245278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.245232    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"]
Oct 30 22:37:46.246125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246098    5028 cpu_manager.go:414] "containerMap: removing container" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy"
Oct 30 22:37:46.246125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246124    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy"
Oct 30 22:37:46.246306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246165    5028 cpu_manager.go:414] "containerMap: removing container" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy"
Oct 30 22:37:46.246306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246174    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy"
Oct 30 22:37:46.246306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246228    5028 cpu_manager.go:414] "containerMap: removing container" podUID="188ff58b-dfa2-4a64-9135-616e23d35641" containerName="revoke-keys"
Oct 30 22:37:46.246306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246236    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188ff58b-dfa2-4a64-9135-616e23d35641" containerName="revoke-keys"
Oct 30 22:37:46.246556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246407    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="188ff58b-dfa2-4a64-9135-616e23d35641" containerName="revoke-keys"
Oct 30 22:37:46.246556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246428    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy"
Oct 30 22:37:46.246556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.246444    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" containerName="istio-proxy"
Oct 30 22:37:46.252068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.252042    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.275171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.275136    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"]
Oct 30 22:37:46.279613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.279580    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6116bd-openshift-default-6c77499645-jnlgk"]
Oct 30 22:37:46.302103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.302056    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:46.302103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.302081    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:46.302103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.302083    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:46.302724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.302702    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:37:46.303580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.303562    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"]
Oct 30 22:37:46.387107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.387013    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678xn\" (UniqueName: \"kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.387107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.387050    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.387319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.387146    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.488265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.488231    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.488457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.488441    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678xn\" (UniqueName: \"kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.488533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.488468    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.489158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.489137    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.490594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.490575    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.528644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.528605    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 22:37:46.528875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.528680    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 22:37:46.558723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.558434    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-678xn\" (UniqueName: \"kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn\") pod \"maas-api-revoke-keys-e2e-api-b-b5313f-nf66h\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.570122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.570072    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:46.841769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.841724    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:46.983065 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:46.983036    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93c866f_db20_4126_a3f2_a0c6f1e4e8a8.slice/crio-ba9516aeccccdb6f263d6eb5ac42a0e2e897a5cc793c6b2a860038d20b0430df WatchSource:0}: Error finding container ba9516aeccccdb6f263d6eb5ac42a0e2e897a5cc793c6b2a860038d20b0430df: Status 404 returned error can't find the container with id ba9516aeccccdb6f263d6eb5ac42a0e2e897a5cc793c6b2a860038d20b0430df
Oct 30 22:37:46.983406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.983385    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"]
Oct 30 22:37:46.999850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:46.999826    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:47.118660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:47.118613    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h" event={"ID":"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8","Type":"ContainerStarted","Data":"ba9516aeccccdb6f263d6eb5ac42a0e2e897a5cc793c6b2a860038d20b0430df"}
Oct 30 22:37:47.213759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:47.213680    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:47.377938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:47.363667    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b31fa1-9460-47b4-bddc-080508b9c5b7" path="/var/lib/kubelet/pods/74b31fa1-9460-47b4-bddc-080508b9c5b7/volumes"
Oct 30 22:37:47.534178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:47.534103    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:47.534629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:47.534605    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:47.535528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:47.535507    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:37:48.080767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.080736    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:48.141676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.141634    5028 generic.go:356] "Generic (PLEG): container finished" podID="b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" containerID="ebaae30b26137607f4459fc873068151c624e13f68456766a978712c28726f61" exitCode=0
Oct 30 22:37:48.141849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.141718    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h" event={"ID":"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8","Type":"ContainerDied","Data":"ebaae30b26137607f4459fc873068151c624e13f68456766a978712c28726f61"}
Oct 30 22:37:48.630027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.629979    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"]
Oct 30 22:37:48.648214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.648180    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.678709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.657965    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.680833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.680807    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.681087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.681068    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.681338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.681323    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdp4t\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.681475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.681462    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.690745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.681677    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.691091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.691056    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.694738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.691273    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.714002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.713968    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.799080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.796235    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:48.799532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.799506    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-771ff9-openshift-default-dockercfg-9zgs5\""
Oct 30 22:37:48.812189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.811797    5028 patch_prober.go:29] interesting pod/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:37:48.812189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.811859    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:37:48.840753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.840720    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.841378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.841264    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.841525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.841348    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.841525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.841457    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.841525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.841496    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.841803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.841781    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.842613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.842559    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.843305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.843284    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdp4t\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.843740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.843720    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.844370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.844306    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.844702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.844650    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.844787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.844756    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.844917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.844817    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.845201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.845182    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.845314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.845204    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.846834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.846808    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.848331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.848296    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"]
Oct 30 22:37:48.902158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.902068    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:48.903014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:48.902990    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdp4t\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t\") pod \"e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") " pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:49.054909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.054860    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:49.188843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.188809    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:49.361957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.361924    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:49.463159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.463122    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn\" (UniqueName: \"kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn\") pod \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") "
Oct 30 22:37:49.463336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.463183    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca\") pod \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") "
Oct 30 22:37:49.463423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.463352    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca\") pod \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\" (UID: \"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8\") "
Oct 30 22:37:49.463814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.463781    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca" pod "b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" (UID: "b93c866f-db20-4126-a3f2-a0c6f1e4e8a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:49.463978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.463961    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.465490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.465460    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca" pod "b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" (UID: "b93c866f-db20-4126-a3f2-a0c6f1e4e8a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:49.465609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.465474    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn" pod "b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" (UID: "b93c866f-db20-4126-a3f2-a0c6f1e4e8a8"). InnerVolumeSpecName "kube-api-access-678xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:49.486730 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:49.486664    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832c3337_d99c_4cbd_ba83_fbfa6d2ac0de.slice/crio-abffdec49078abb84261cbc2e9c1e69ec1fd62be7cf4a364c04dc56587c737fa WatchSource:0}: Error finding container abffdec49078abb84261cbc2e9c1e69ec1fd62be7cf4a364c04dc56587c737fa: Status 404 returned error can't find the container with id abffdec49078abb84261cbc2e9c1e69ec1fd62be7cf4a364c04dc56587c737fa
Oct 30 22:37:49.487771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.487613    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"]
Oct 30 22:37:49.490778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.490739    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:49.490862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.490824    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:49.490901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.490868    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:49.565503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.565470    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-678xn\" (UniqueName: \"kubernetes.io/projected/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-kube-api-access-678xn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.565503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.565502    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.648708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:49.648636    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.195440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.195358    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"
Oct 30 22:37:50.195440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.195411    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h" event={"ID":"b93c866f-db20-4126-a3f2-a0c6f1e4e8a8","Type":"ContainerDied","Data":"ba9516aeccccdb6f263d6eb5ac42a0e2e897a5cc793c6b2a860038d20b0430df"}
Oct 30 22:37:50.195635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.195491    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9516aeccccdb6f263d6eb5ac42a0e2e897a5cc793c6b2a860038d20b0430df"
Oct 30 22:37:50.197630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.197597    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" event={"ID":"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de","Type":"ContainerStarted","Data":"89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"}
Oct 30 22:37:50.197761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.197639    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" event={"ID":"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de","Type":"ContainerStarted","Data":"abffdec49078abb84261cbc2e9c1e69ec1fd62be7cf4a364c04dc56587c737fa"}
Oct 30 22:37:50.319738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.319665    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podStartSLOduration=2.31964901 podStartE2EDuration="2.31964901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:50.315495093 +0000 UTC m=+2201.165008066" watchObservedRunningTime="2026-10-30 22:37:50.31964901 +0000 UTC m=+2201.169162051"
Oct 30 22:37:50.740461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.740371    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.786647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.786621    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:37:50.883062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883032    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883085    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883113    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883133    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883153    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883256    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883317    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883397    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883525    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:50.883554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883543    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:50.883872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883552    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:50.883872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883571    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket\") pod \"256a1079-f087-481a-98ad-745f95415d8f\" (UID: \"256a1079-f087-481a-98ad-745f95415d8f\") "
Oct 30 22:37:50.883872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883745    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:50.883872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.883765    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:50.884203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.884183    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256a1079-f087-481a-98ad-745f95415d8f-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.884272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.884204    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.884272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.884214    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.884272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.884222    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.884272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.884232    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.885556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.885533    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:37:50.885669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.885625    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "kube-api-access-2mlpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:50.885807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.885787    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:50.885877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.885845    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token" pod "256a1079-f087-481a-98ad-745f95415d8f" (UID: "256a1079-f087-481a-98ad-745f95415d8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:50.909078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.909046    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.985951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.985912    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlpr\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-kube-api-access-2mlpr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.985951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.985949    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256a1079-f087-481a-98ad-745f95415d8f-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.985951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.985960    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256a1079-f087-481a-98ad-745f95415d8f-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.986195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:50.985969    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256a1079-f087-481a-98ad-745f95415d8f-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:51.055272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.055178    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:51.061551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.061520    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:51.206958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.206925    5028 generic.go:356] "Generic (PLEG): container finished" podID="256a1079-f087-481a-98ad-745f95415d8f" containerID="478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31" exitCode=0
Oct 30 22:37:51.207140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.206973    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"
Oct 30 22:37:51.207140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.207014    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" event={"ID":"256a1079-f087-481a-98ad-745f95415d8f","Type":"ContainerDied","Data":"478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31"}
Oct 30 22:37:51.207140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.207071    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6" event={"ID":"256a1079-f087-481a-98ad-745f95415d8f","Type":"ContainerDied","Data":"5538332c39d151e6b27ed247308461fb494b54b2e2e25ad9dd7a7f180d9a57ef"}
Oct 30 22:37:51.207140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.207103    5028 scope.go:122] "RemoveContainer" containerID="478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31"
Oct 30 22:37:51.207480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.207460    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:51.208835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.208813    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:37:51.220838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.220817    5028 scope.go:122] "RemoveContainer" containerID="9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"
Oct 30 22:37:51.232817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.232790    5028 scope.go:122] "RemoveContainer" containerID="478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31"
Oct 30 22:37:51.233190 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:51.233164    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31\": container with ID starting with 478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31 not found: ID does not exist" containerID="478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31"
Oct 30 22:37:51.233256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.233203    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31"} err="failed to get container status \"478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31\": rpc error: code = NotFound desc = could not find container \"478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31\": container with ID starting with 478f0918d4767cfce18f6a6a38f8363ee520c371732efed3fe52b8ba8fe47b31 not found: ID does not exist"
Oct 30 22:37:51.233256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.233222    5028 scope.go:122] "RemoveContainer" containerID="9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"
Oct 30 22:37:51.233558 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:51.233525    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97\": container with ID starting with 9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97 not found: ID does not exist" containerID="9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"
Oct 30 22:37:51.233632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.233569    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97"} err="failed to get container status \"9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97\": rpc error: code = NotFound desc = could not find container \"9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97\": container with ID starting with 9ee0cc6b5af3f9a22897de250d6f375d0879043848b74483504e5ca6e4bafa97 not found: ID does not exist"
Oct 30 22:37:51.293003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.292974    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"]
Oct 30 22:37:51.396756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.396659    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"]
Oct 30 22:37:51.484068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.484027    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"]
Oct 30 22:37:51.493011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.492978    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a3600a-openshift-default-5d865f68dc-fjcx6"]
Oct 30 22:37:51.635652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.635616    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:51.794300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:51.794261    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:52.501901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:52.501864    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:52.855772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:52.855353    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.294320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:53.294284    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256a1079-f087-481a-98ad-745f95415d8f" path="/var/lib/kubelet/pods/256a1079-f087-481a-98ad-745f95415d8f/volumes"
Oct 30 22:37:53.392332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:53.392296    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.596829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:53.596744    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:53.756737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:53.756696    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:54.000438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.000403    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"]
Oct 30 22:37:54.001119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001103    5028 cpu_manager.go:414] "containerMap: removing container" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy"
Oct 30 22:37:54.001177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001120    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy"
Oct 30 22:37:54.001177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001158    5028 cpu_manager.go:414] "containerMap: removing container" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy"
Oct 30 22:37:54.001177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001163    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy"
Oct 30 22:37:54.001285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001178    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" containerName="revoke-keys"
Oct 30 22:37:54.001285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001184    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" containerName="revoke-keys"
Oct 30 22:37:54.001353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001299    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy"
Oct 30 22:37:54.001353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001312    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="256a1079-f087-481a-98ad-745f95415d8f" containerName="istio-proxy"
Oct 30 22:37:54.001353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.001319    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" containerName="revoke-keys"
Oct 30 22:37:54.006781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.006756    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068285    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068420    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5jn\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068457    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068491    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068621    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068706    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068786    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.068936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068891    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.069111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.068966    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.117806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.117763    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"]
Oct 30 22:37:54.170586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170545    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.170845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170607    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.170845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170658    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.170845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170744    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5jn\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.170845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170763    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.170845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170798    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.170845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170847    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.170880    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.171102    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.171230    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.171336    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.171423    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.171471    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.171807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.171789    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.173246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.173224    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.173339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.173287    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.230194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.230152    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.230383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.230351    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5jn\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.324185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.324086    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:54.773328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.773292    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"]
Oct 30 22:37:54.797634 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:54.797604    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45f0344_ae7c_471a_9f98_cbde455efcda.slice/crio-32ea90757261a4e9c3e8616f06d3cd41e3ad4241dcbed349d248000fcb464dc9 WatchSource:0}: Error finding container 32ea90757261a4e9c3e8616f06d3cd41e3ad4241dcbed349d248000fcb464dc9: Status 404 returned error can't find the container with id 32ea90757261a4e9c3e8616f06d3cd41e3ad4241dcbed349d248000fcb464dc9
Oct 30 22:37:54.800195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.800160    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:54.800255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.800230    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:54.800299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.800262    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:37:54.849114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:54.849056    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:55.264279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:55.264236    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" event={"ID":"b45f0344-ae7c-471a-9f98-cbde455efcda","Type":"ContainerStarted","Data":"0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d"}
Oct 30 22:37:55.264764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:55.264309    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" event={"ID":"b45f0344-ae7c-471a-9f98-cbde455efcda","Type":"ContainerStarted","Data":"32ea90757261a4e9c3e8616f06d3cd41e3ad4241dcbed349d248000fcb464dc9"}
Oct 30 22:37:55.324247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:55.324212    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:37:55.326075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:55.326029    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:37:55.326243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:55.326099    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:37:55.382519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:55.382462    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podStartSLOduration=2.382447232 podStartE2EDuration="2.382447232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:55.379639772 +0000 UTC m=+2206.229152748" watchObservedRunningTime="2026-10-30 22:37:55.382447232 +0000 UTC m=+2206.231960205"
Oct 30 22:37:56.315218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:56.314815    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n" podUID="aa019642-4b4e-43fb-9b47-f83f3744f656" containerName="payload-processing" containerID="cri-o://a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853" gracePeriod=25
Oct 30 22:37:56.328892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:56.328858    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:37:56.329189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:56.329145    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:37:56.413710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:56.407780    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz" podUID="00b20f1e-4231-4cf8-9cdb-2f1191307323" containerName="payload-pre-processing" containerID="cri-o://fb30449ede5f938f5db3fd814838de79274522087ed7a46414f247d75f08350f" gracePeriod=25
Oct 30 22:37:56.561932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:56.561730    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:56.858762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:56.851799    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:57.393456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.389344    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:37:57.393456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.389393    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:37:57.408847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.408814    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"]
Oct 30 22:37:57.411144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.411115    5028 generic.go:356] "Generic (PLEG): container finished" podID="00b20f1e-4231-4cf8-9cdb-2f1191307323" containerID="fb30449ede5f938f5db3fd814838de79274522087ed7a46414f247d75f08350f" exitCode=0
Oct 30 22:37:57.411735 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.411708    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:37:57.412952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.412931    5028 generic.go:356] "Generic (PLEG): container finished" podID="aa019642-4b4e-43fb-9b47-f83f3744f656" containerID="a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853" exitCode=0
Oct 30 22:37:57.413906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.413879    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"]
Oct 30 22:37:57.414175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.414139    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz" event={"ID":"00b20f1e-4231-4cf8-9cdb-2f1191307323","Type":"ContainerDied","Data":"fb30449ede5f938f5db3fd814838de79274522087ed7a46414f247d75f08350f"}
Oct 30 22:37:57.414291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.414192    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"]
Oct 30 22:37:57.442611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.414086    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.442611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.415718    5028 cpu_manager.go:414] "containerMap: removing container" podUID="aa019642-4b4e-43fb-9b47-f83f3744f656" containerName="payload-processing"
Oct 30 22:37:57.442611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.415736    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa019642-4b4e-43fb-9b47-f83f3744f656" containerName="payload-processing"
Oct 30 22:37:57.442611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.415980    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa019642-4b4e-43fb-9b47-f83f3744f656" containerName="payload-processing"
Oct 30 22:37:57.456743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.452909    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n" event={"ID":"aa019642-4b4e-43fb-9b47-f83f3744f656","Type":"ContainerDied","Data":"a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853"}
Oct 30 22:37:57.456743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.453076    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"]
Oct 30 22:37:57.456743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.453113    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n" event={"ID":"aa019642-4b4e-43fb-9b47-f83f3744f656","Type":"ContainerDied","Data":"e8f7162594d80d9dc466796b60ac1fc5023a40bea50e701ff9208e7f9b32458c"}
Oct 30 22:37:57.456743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.453149    5028 scope.go:122] "RemoveContainer" containerID="a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853"
Oct 30 22:37:57.456743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.453315    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.474621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.465634    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-771ff9\""
Oct 30 22:37:57.489002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.487866    5028 scope.go:122] "RemoveContainer" containerID="a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853"
Oct 30 22:37:57.489002 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:57.488217    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853\": container with ID starting with a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853 not found: ID does not exist" containerID="a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853"
Oct 30 22:37:57.489002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.488253    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853"} err="failed to get container status \"a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853\": rpc error: code = NotFound desc = could not find container \"a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853\": container with ID starting with a2d6a59bedf90235991fb921051de2ddac0e5d6f6af435d72d6295eb93b37853 not found: ID does not exist"
Oct 30 22:37:57.495713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.490801    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-771ff9-dockercfg-8mfv5\""
Oct 30 22:37:57.560875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.560286    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume\") pod \"aa019642-4b4e-43fb-9b47-f83f3744f656\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") "
Oct 30 22:37:57.560875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.560427    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m\" (UniqueName: \"kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m\") pod \"aa019642-4b4e-43fb-9b47-f83f3744f656\" (UID: \"aa019642-4b4e-43fb-9b47-f83f3744f656\") "
Oct 30 22:37:57.561772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.561666    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume" pod "aa019642-4b4e-43fb-9b47-f83f3744f656" (UID: "aa019642-4b4e-43fb-9b47-f83f3744f656"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:57.570734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.564526    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m" pod "aa019642-4b4e-43fb-9b47-f83f3744f656" (UID: "aa019642-4b4e-43fb-9b47-f83f3744f656"). InnerVolumeSpecName "kube-api-access-r6p9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:57.571262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.571234    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.576976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.576287    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hngf\" (UniqueName: \"kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf\") pod \"payload-processing-e2e-api-a-771ff9-65d554b547-ct57x\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") " pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.576976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.576459    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume\") pod \"payload-processing-e2e-api-a-771ff9-65d554b547-ct57x\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") " pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.576976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.576488    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlls\" (UniqueName: \"kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls\") pod \"payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.576976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.576627    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa019642-4b4e-43fb-9b47-f83f3744f656-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:57.576976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.576644    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6p9m\" (UniqueName: \"kubernetes.io/projected/aa019642-4b4e-43fb-9b47-f83f3744f656-kube-api-access-r6p9m\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:57.595434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.595395    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:37:57.677438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.677401    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.677642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.677462    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hngf\" (UniqueName: \"kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf\") pod \"payload-processing-e2e-api-a-771ff9-65d554b547-ct57x\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") " pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.677642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.677533    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume\") pod \"payload-processing-e2e-api-a-771ff9-65d554b547-ct57x\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") " pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.677808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.677785    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlls\" (UniqueName: \"kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls\") pod \"payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.678363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.678342    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume\") pod \"payload-processing-e2e-api-a-771ff9-65d554b547-ct57x\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") " pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.678439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.678414    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.746407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.746364    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hngf\" (UniqueName: \"kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf\") pod \"payload-processing-e2e-api-a-771ff9-65d554b547-ct57x\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") " pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.750891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.750282    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlls\" (UniqueName: \"kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls\") pod \"payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.778468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.778435    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b\" (UniqueName: \"kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b\") pod \"00b20f1e-4231-4cf8-9cdb-2f1191307323\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") "
Oct 30 22:37:57.778672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.778618    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume\") pod \"00b20f1e-4231-4cf8-9cdb-2f1191307323\" (UID: \"00b20f1e-4231-4cf8-9cdb-2f1191307323\") "
Oct 30 22:37:57.779124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.779088    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume" pod "00b20f1e-4231-4cf8-9cdb-2f1191307323" (UID: "00b20f1e-4231-4cf8-9cdb-2f1191307323"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:57.780945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.780914    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b" pod "00b20f1e-4231-4cf8-9cdb-2f1191307323" (UID: "00b20f1e-4231-4cf8-9cdb-2f1191307323"). InnerVolumeSpecName "kube-api-access-gfb2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:57.783149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.783127    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:57.794469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.794445    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:57.882883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.882852    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfb2b\" (UniqueName: \"kubernetes.io/projected/00b20f1e-4231-4cf8-9cdb-2f1191307323-kube-api-access-gfb2b\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:57.882883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:57.882882    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b20f1e-4231-4cf8-9cdb-2f1191307323-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:58.279952 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:58.279157    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8effebf9_048b_4365_80ed_f9cb50d2d6b9.slice/crio-72c66b2760dff2df93346edda971fed578b9a35c313f5844162bd74733b08686 WatchSource:0}: Error finding container 72c66b2760dff2df93346edda971fed578b9a35c313f5844162bd74733b08686: Status 404 returned error can't find the container with id 72c66b2760dff2df93346edda971fed578b9a35c313f5844162bd74733b08686
Oct 30 22:37:58.282399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.282372    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"]
Oct 30 22:37:58.329379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.329251    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:37:58.329379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.329312    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:37:58.349648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.349611    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"]
Oct 30 22:37:58.371378 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:37:58.371348    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638522ab_0274_4468_9e3f_ed8e8219a080.slice/crio-5c3a1a0d3f8225af29b9cf7b424ece38c66cabc3d0a4e523f3a2612272fe8b57 WatchSource:0}: Error finding container 5c3a1a0d3f8225af29b9cf7b424ece38c66cabc3d0a4e523f3a2612272fe8b57: Status 404 returned error can't find the container with id 5c3a1a0d3f8225af29b9cf7b424ece38c66cabc3d0a4e523f3a2612272fe8b57
Oct 30 22:37:58.458303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.457795    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj" event={"ID":"8effebf9-048b-4365-80ed-f9cb50d2d6b9","Type":"ContainerStarted","Data":"72c66b2760dff2df93346edda971fed578b9a35c313f5844162bd74733b08686"}
Oct 30 22:37:58.461759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.461366    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz" event={"ID":"00b20f1e-4231-4cf8-9cdb-2f1191307323","Type":"ContainerDied","Data":"400e3a56744323223c2cb0e2d190126f3e1f1aa212e0c0d24f7948f846698195"}
Oct 30 22:37:58.461759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.461434    5028 scope.go:122] "RemoveContainer" containerID="fb30449ede5f938f5db3fd814838de79274522087ed7a46414f247d75f08350f"
Oct 30 22:37:58.461759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.461643    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"
Oct 30 22:37:58.471580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.471133    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"
Oct 30 22:37:58.479542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.479380    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x" event={"ID":"638522ab-0274-4468-9e3f-ed8e8219a080","Type":"ContainerStarted","Data":"5c3a1a0d3f8225af29b9cf7b424ece38c66cabc3d0a4e523f3a2612272fe8b57"}
Oct 30 22:37:58.485613 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:37:58.485571    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:37:58.646314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.646213    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"]
Oct 30 22:37:58.665500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.665459    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29bbe735-859dbf5fc-ck47n"]
Oct 30 22:37:58.751392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.751365    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"]
Oct 30 22:37:58.755937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:58.755912    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29bbe735-7d7b774575-tclzz"]
Oct 30 22:37:59.295430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.295395    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b20f1e-4231-4cf8-9cdb-2f1191307323" path="/var/lib/kubelet/pods/00b20f1e-4231-4cf8-9cdb-2f1191307323/volumes"
Oct 30 22:37:59.295891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.295877    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa019642-4b4e-43fb-9b47-f83f3744f656" path="/var/lib/kubelet/pods/aa019642-4b4e-43fb-9b47-f83f3744f656/volumes"
Oct 30 22:37:59.336717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.331013    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:37:59.337039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.336999    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:37:59.489982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.489949    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x" event={"ID":"638522ab-0274-4468-9e3f-ed8e8219a080","Type":"ContainerStarted","Data":"5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09"}
Oct 30 22:37:59.490415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.490270    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:37:59.491917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.491889    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj" event={"ID":"8effebf9-048b-4365-80ed-f9cb50d2d6b9","Type":"ContainerStarted","Data":"178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8"}
Oct 30 22:37:59.492070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.491941    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:37:59.607953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.607870    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:59.633747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.633676    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x" podStartSLOduration=2.633659506 podStartE2EDuration="2.633659506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:59.632086787 +0000 UTC m=+2210.481599757" watchObservedRunningTime="2026-10-30 22:37:59.633659506 +0000 UTC m=+2210.483172479"
Oct 30 22:37:59.751799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:37:59.751724    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj" podStartSLOduration=2.751702413 podStartE2EDuration="2.751702413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:59.749140059 +0000 UTC m=+2210.598653055" watchObservedRunningTime="2026-10-30 22:37:59.751702413 +0000 UTC m=+2210.601215381"
Oct 30 22:38:00.070981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:00.070841    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"]
Oct 30 22:38:00.076866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:00.076830    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4234c5fa-cbqh8"]
Oct 30 22:38:00.332773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:00.325114    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:38:00.332773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:00.325188    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:38:00.593976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:00.591661    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:00.858795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:00.836493    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:01.292955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:01.292918    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a373be5d-437b-4710-8ddc-9153354d1089" path="/var/lib/kubelet/pods/a373be5d-437b-4710-8ddc-9153354d1089/volumes"
Oct 30 22:38:01.347714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:01.343820    5028 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 22:38:01.347714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:01.343884    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 22:38:02.449709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:02.449273    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:38:02.450854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:02.450581    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:38:02.452751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:02.452726    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:38:04.206096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:04.206054    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:08.742830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:08.742782    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:09.613890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:09.613856    5028 generic.go:356] "Generic (PLEG): container finished" podID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerID="6cf6c97880eb6322f3dc2dde625faca51dbf9e062438f8630355dd97b306f46c" exitCode=0
Oct 30 22:38:09.614091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:09.613940    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" event={"ID":"d91a055c-7859-4bf5-8e64-25d29b12a7af","Type":"ContainerDied","Data":"6cf6c97880eb6322f3dc2dde625faca51dbf9e062438f8630355dd97b306f46c"}
Oct 30 22:38:09.614421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:09.614405    5028 scope.go:122] "RemoveContainer" containerID="6cf6c97880eb6322f3dc2dde625faca51dbf9e062438f8630355dd97b306f46c"
Oct 30 22:38:09.616020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:09.615990    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:38:09.616108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:09.616064    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:38:09.616108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:09.616099    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:38:10.506302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:10.506269    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:38:10.506775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:10.506346    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:38:10.665259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:10.665228    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" event={"ID":"d91a055c-7859-4bf5-8e64-25d29b12a7af","Type":"ContainerStarted","Data":"84e955f10d58ee87344fb5859212ca863d792ee746d30efb33d07e4dcd8210dd"}
Oct 30 22:38:11.092552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:11.092517    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:38:11.093753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:11.093721    5028 patch_prober.go:29] interesting pod/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:38:11.093903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:11.093772    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 22:38:12.093135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:12.093097    5028 patch_prober.go:29] interesting pod/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:38:12.093575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:12.093155    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 22:38:12.524712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:12.516764    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:13.093154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:13.093114    5028 patch_prober.go:29] interesting pod/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:38:13.093577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:13.093183    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 22:38:13.246577 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:13.246538    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:14.092863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:14.092819    5028 patch_prober.go:29] interesting pod/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:38:14.093091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:14.092892    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 22:38:14.302451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:14.302417    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:15.093472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:15.093441    5028 patch_prober.go:29] interesting pod/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:38:15.093669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:15.093490    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 22:38:15.148346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:15.148311    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:38:16.098285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:16.098249    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:38:16.099416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:16.099394    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:38:17.190963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:17.190924    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:17.588451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:17.588420    5028 scope.go:122] "RemoveContainer" containerID="881833bb02281404fdb2a4df039b7917898c29466c6ba9cd7d5fa920e1f59d5c"
Oct 30 22:38:18.640099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:18.640064    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:18.868137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:18.868097    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:19.042865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:19.042835    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:19.168248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:19.168208    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"]
Oct 30 22:38:19.220593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:19.220548    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bbd10f-694sr"]
Oct 30 22:38:19.293050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:19.292974    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0fd2ec-a087-4f38-83be-0bb91a8e71d0" path="/var/lib/kubelet/pods/8d0fd2ec-a087-4f38-83be-0bb91a8e71d0/volumes"
Oct 30 22:38:19.625384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:19.625285    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:21.061225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:21.061183    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.697116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:23.697066    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:25.891738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:25.891701    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:26.110272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:26.110241    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:26.244395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:26.244348    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"]
Oct 30 22:38:26.244647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:26.244614    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy" containerID="cri-o://5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55" gracePeriod=30
Oct 30 22:38:26.261002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:26.260966    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:26.462671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:26.462630    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:26.822101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:26.822060    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:27.489544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.489512    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"]
Oct 30 22:38:27.490251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.490235    5028 cpu_manager.go:414] "containerMap: removing container" podUID="00b20f1e-4231-4cf8-9cdb-2f1191307323" containerName="payload-pre-processing"
Oct 30 22:38:27.490307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.490253    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b20f1e-4231-4cf8-9cdb-2f1191307323" containerName="payload-pre-processing"
Oct 30 22:38:27.490400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.490390    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b20f1e-4231-4cf8-9cdb-2f1191307323" containerName="payload-pre-processing"
Oct 30 22:38:27.496122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.496105    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.551212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.551169    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"]
Oct 30 22:38:27.591339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.591293    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:27.591339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.591295    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:27.591339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.591294    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:27.593146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.593124    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:38:27.605726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.605270    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.605726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.605384    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgvxp\" (UniqueName: \"kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.605726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.605544    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.707137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.707098    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.707359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.707189    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgvxp\" (UniqueName: \"kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.707432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.707364    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.707877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.707853    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.709962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.709941    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.771336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.768052    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgvxp\" (UniqueName: \"kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp\") pod \"maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.811808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.811763    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:27.819101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.819069    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:27.993865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:27.989328    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:28.206802 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:38:28.206767    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98d6477_d401_4cb3_ac4e_be00821a7fbe.slice/crio-08db04fdff74f0dcac33113bed1349ebe931824da892499b4ebf66ecd3ff4266 WatchSource:0}: Error finding container 08db04fdff74f0dcac33113bed1349ebe931824da892499b4ebf66ecd3ff4266: Status 404 returned error can't find the container with id 08db04fdff74f0dcac33113bed1349ebe931824da892499b4ebf66ecd3ff4266
Oct 30 22:38:28.209040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:28.209012    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"]
Oct 30 22:38:28.439565 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:38:28.439531    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:38:28.732784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:28.732749    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:28.835003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:28.834961    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4" event={"ID":"c98d6477-d401-4cb3-ac4e-be00821a7fbe","Type":"ContainerStarted","Data":"74a4505eeea9b8106d826894a6f1ca0c3a1c1f9a9517a9ab0b6750d7ff70bbff"}
Oct 30 22:38:28.835204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:28.835030    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4" event={"ID":"c98d6477-d401-4cb3-ac4e-be00821a7fbe","Type":"ContainerStarted","Data":"08db04fdff74f0dcac33113bed1349ebe931824da892499b4ebf66ecd3ff4266"}
Oct 30 22:38:28.947885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:28.947772    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4" podStartSLOduration=1.9477581370000001 podStartE2EDuration="1.947758137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:28.946749429 +0000 UTC m=+2239.796262398" watchObservedRunningTime="2026-10-30 22:38:28.947758137 +0000 UTC m=+2239.797271144"
Oct 30 22:38:29.845949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:29.845909    5028 generic.go:356] "Generic (PLEG): container finished" podID="c98d6477-d401-4cb3-ac4e-be00821a7fbe" containerID="74a4505eeea9b8106d826894a6f1ca0c3a1c1f9a9517a9ab0b6750d7ff70bbff" exitCode=0
Oct 30 22:38:29.846362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:29.845996    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4" event={"ID":"c98d6477-d401-4cb3-ac4e-be00821a7fbe","Type":"ContainerDied","Data":"74a4505eeea9b8106d826894a6f1ca0c3a1c1f9a9517a9ab0b6750d7ff70bbff"}
Oct 30 22:38:31.024429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.024399    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:31.157012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.156911    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca\") pod \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") "
Oct 30 22:38:31.157201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.157037    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp\" (UniqueName: \"kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp\") pod \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") "
Oct 30 22:38:31.157201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.157182    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca\") pod \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\" (UID: \"c98d6477-d401-4cb3-ac4e-be00821a7fbe\") "
Oct 30 22:38:31.157771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.157734    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca" pod "c98d6477-d401-4cb3-ac4e-be00821a7fbe" (UID: "c98d6477-d401-4cb3-ac4e-be00821a7fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:31.157927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.157896    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.160409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.160380    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca" pod "c98d6477-d401-4cb3-ac4e-be00821a7fbe" (UID: "c98d6477-d401-4cb3-ac4e-be00821a7fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:31.160871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.160847    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp" pod "c98d6477-d401-4cb3-ac4e-be00821a7fbe" (UID: "c98d6477-d401-4cb3-ac4e-be00821a7fbe"). InnerVolumeSpecName "kube-api-access-fgvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:31.260089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.260047    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgvxp\" (UniqueName: \"kubernetes.io/projected/c98d6477-d401-4cb3-ac4e-be00821a7fbe-kube-api-access-fgvxp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.260089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.260092    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98d6477-d401-4cb3-ac4e-be00821a7fbe-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.511783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.511748    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:38:31.564502 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564470    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.564752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564524    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.564752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564594    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.564752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564611    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.564752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564665    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.564752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564752    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.565076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564811    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.565076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564875    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.565076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564892    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:31.565076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.564952    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy\") pod \"f6de087b-d027-4356-a2a2-4e55b5a258cf\" (UID: \"f6de087b-d027-4356-a2a2-4e55b5a258cf\") "
Oct 30 22:38:31.565303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565099    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:31.565303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565124    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:31.565303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565159    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:31.565481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565304    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:31.565758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565732    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.565902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565762    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.565902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565777    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de087b-d027-4356-a2a2-4e55b5a258cf-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.565902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565792    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.565902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.565806    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.567088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.567059    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "kube-api-access-5z6wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:31.567334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.567307    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:31.567427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.567367    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:31.567427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.567390    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token" pod "f6de087b-d027-4356-a2a2-4e55b5a258cf" (UID: "f6de087b-d027-4356-a2a2-4e55b5a258cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:31.667136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.667100    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.667136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.667134    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.667359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.667146    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6wb\" (UniqueName: \"kubernetes.io/projected/f6de087b-d027-4356-a2a2-4e55b5a258cf-kube-api-access-5z6wb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.667359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.667157    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de087b-d027-4356-a2a2-4e55b5a258cf-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:31.800026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.799980    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:31.868338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.868306    5028 generic.go:356] "Generic (PLEG): container finished" podID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerID="5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55" exitCode=0
Oct 30 22:38:31.868541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.868365    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"
Oct 30 22:38:31.868541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.868405    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" event={"ID":"f6de087b-d027-4356-a2a2-4e55b5a258cf","Type":"ContainerDied","Data":"5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55"}
Oct 30 22:38:31.868541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.868446    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q" event={"ID":"f6de087b-d027-4356-a2a2-4e55b5a258cf","Type":"ContainerDied","Data":"354d954016a34ec2b33ace6c5fa6ea2e689bedce6b259f9573ccbd6e638a9ce5"}
Oct 30 22:38:31.868541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.868479    5028 scope.go:122] "RemoveContainer" containerID="5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55"
Oct 30 22:38:31.870198 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.870177    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"
Oct 30 22:38:31.870310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.870203    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4" event={"ID":"c98d6477-d401-4cb3-ac4e-be00821a7fbe","Type":"ContainerDied","Data":"08db04fdff74f0dcac33113bed1349ebe931824da892499b4ebf66ecd3ff4266"}
Oct 30 22:38:31.870310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.870256    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08db04fdff74f0dcac33113bed1349ebe931824da892499b4ebf66ecd3ff4266"
Oct 30 22:38:31.883575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.883552    5028 scope.go:122] "RemoveContainer" containerID="7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"
Oct 30 22:38:31.895782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.895757    5028 scope.go:122] "RemoveContainer" containerID="5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55"
Oct 30 22:38:31.896123 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:38:31.896093    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55\": container with ID starting with 5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55 not found: ID does not exist" containerID="5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55"
Oct 30 22:38:31.896225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.896137    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55"} err="failed to get container status \"5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55\": rpc error: code = NotFound desc = could not find container \"5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55\": container with ID starting with 5b71d7e5d8ac1d1bdd2a2da9d0b62031ea368b25bb18db4a7d2d270904032e55 not found: ID does not exist"
Oct 30 22:38:31.896225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.896165    5028 scope.go:122] "RemoveContainer" containerID="7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"
Oct 30 22:38:31.896498 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:38:31.896472    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac\": container with ID starting with 7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac not found: ID does not exist" containerID="7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"
Oct 30 22:38:31.896573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.896508    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac"} err="failed to get container status \"7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac\": rpc error: code = NotFound desc = could not find container \"7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac\": container with ID starting with 7defd30ec902ef44be2abb036aa5d3f5d367002e2186acd1b0d7f8343300faac not found: ID does not exist"
Oct 30 22:38:31.965292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:31.965248    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:32.031226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:32.031188    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"]
Oct 30 22:38:32.037372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:32.037335    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-bbd10f-openshift-default-6fd95cd469-vzr5q"]
Oct 30 22:38:32.126358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:32.126265    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:33.293239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:33.293204    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" path="/var/lib/kubelet/pods/f6de087b-d027-4356-a2a2-4e55b5a258cf/volumes"
Oct 30 22:38:33.586365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:33.586266    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:36.682294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:36.682253    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:36.917468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:36.917424    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:37.076295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.076258    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"]
Oct 30 22:38:37.077487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077464    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy"
Oct 30 22:38:37.077557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077491    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy"
Oct 30 22:38:37.077557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077509    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c98d6477-d401-4cb3-ac4e-be00821a7fbe" containerName="revoke-keys"
Oct 30 22:38:37.077557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077517    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98d6477-d401-4cb3-ac4e-be00821a7fbe" containerName="revoke-keys"
Oct 30 22:38:37.077557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077528    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy"
Oct 30 22:38:37.077557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077536    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy"
Oct 30 22:38:37.077744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077731    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy"
Oct 30 22:38:37.077782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077748    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98d6477-d401-4cb3-ac4e-be00821a7fbe" containerName="revoke-keys"
Oct 30 22:38:37.077782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.077757    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6de087b-d027-4356-a2a2-4e55b5a258cf" containerName="istio-proxy"
Oct 30 22:38:37.082633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.082606    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.133902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.133866    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"]
Oct 30 22:38:37.174861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.174824    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d585d2\""
Oct 30 22:38:37.192202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.192162    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.192574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.192548    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2fq\" (UniqueName: \"kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq\") pod \"payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.237346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.237099    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"]
Oct 30 22:38:37.259996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.259964    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.295261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.293763    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.295261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.293847    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2fq\" (UniqueName: \"kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq\") pod \"payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.295506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.295304    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.327755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.327562    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d585d2-dockercfg-6ddqv\""
Oct 30 22:38:37.383062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.383017    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"]
Oct 30 22:38:37.393479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.393446    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2fq\" (UniqueName: \"kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq\") pod \"payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.402987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.402950    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:37.404442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.404406    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2gj\" (UniqueName: \"kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj\") pod \"payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.404581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.404484    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.507854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.507494    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2gj\" (UniqueName: \"kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj\") pod \"payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.507854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.507553    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.508254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.508233    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.674717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.658190    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2gj\" (UniqueName: \"kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj\") pod \"payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:37.700727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.676180    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:37.856994 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.856801    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"]
Oct 30 22:38:37.858766 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:38:37.858418    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d1bfc4_14e7_4869_9c41_4b9c17dcb4a1.slice/crio-c25bad7f0990b0c29f1ae3d9b5379f1e35bcf15ce5c2a1edb618a8b5bb29128f WatchSource:0}: Error finding container c25bad7f0990b0c29f1ae3d9b5379f1e35bcf15ce5c2a1edb618a8b5bb29128f: Status 404 returned error can't find the container with id c25bad7f0990b0c29f1ae3d9b5379f1e35bcf15ce5c2a1edb618a8b5bb29128f
Oct 30 22:38:37.924421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.924387    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" event={"ID":"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1","Type":"ContainerStarted","Data":"c25bad7f0990b0c29f1ae3d9b5379f1e35bcf15ce5c2a1edb618a8b5bb29128f"}
Oct 30 22:38:37.949891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:37.949856    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:38.386830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:38.386799    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"]
Oct 30 22:38:38.390054 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:38:38.390023    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bde795e_1061_46dc_bdd7_71f5a27dede1.slice/crio-74624416e9c7699c20cbf57c8c9bc132ae51a448dacd0bbe1e366398c269209b WatchSource:0}: Error finding container 74624416e9c7699c20cbf57c8c9bc132ae51a448dacd0bbe1e366398c269209b: Status 404 returned error can't find the container with id 74624416e9c7699c20cbf57c8c9bc132ae51a448dacd0bbe1e366398c269209b
Oct 30 22:38:38.935787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:38.935753    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" event={"ID":"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1","Type":"ContainerStarted","Data":"1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283"}
Oct 30 22:38:38.936301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:38.936094    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:38.938248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:38.938210    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" event={"ID":"4bde795e-1061-46dc-bdd7-71f5a27dede1","Type":"ContainerStarted","Data":"430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd"}
Oct 30 22:38:38.938412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:38.938265    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" event={"ID":"4bde795e-1061-46dc-bdd7-71f5a27dede1","Type":"ContainerStarted","Data":"74624416e9c7699c20cbf57c8c9bc132ae51a448dacd0bbe1e366398c269209b"}
Oct 30 22:38:38.938501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:38.938481    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:39.050244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.050163    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" podStartSLOduration=2.050146148 podStartE2EDuration="2.050146148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:39.046301791 +0000 UTC m=+2249.895814764" watchObservedRunningTime="2026-10-30 22:38:39.050146148 +0000 UTC m=+2249.899659122"
Oct 30 22:38:39.161890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.161833    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" podStartSLOduration=2.161815544 podStartE2EDuration="2.161815544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:39.156804776 +0000 UTC m=+2250.006317767" watchObservedRunningTime="2026-10-30 22:38:39.161815544 +0000 UTC m=+2250.011328519"
Oct 30 22:38:39.204340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.204252    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:39.468024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.467980    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:39.619029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.618989    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:39.723393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.723294    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"]
Oct 30 22:38:39.730881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.730602    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29bbe735-98l5s"]
Oct 30 22:38:39.771916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:39.771882    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.118594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.118472    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.132107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.132073    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"]
Oct 30 22:38:40.245520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.245481    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"]
Oct 30 22:38:40.278313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.278269    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.486126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.486084    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.674708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.667518    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.759774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.754742    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"]
Oct 30 22:38:40.798986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.798651    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.817217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.817186    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"]
Oct 30 22:38:40.856631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.856593    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-afb98c-openshift-default-dockercfg-2qt2x\""
Oct 30 22:38:40.881117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.881055    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:40.896133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896103    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896163    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896276    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jnw\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896344    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896451    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896748    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896815 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896802    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896843    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.896988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.896973    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.998926    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.998985    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999022    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999110    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999174    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999235    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999267    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jnw\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999314    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:40.999724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999381    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.000368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:40.999946    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.001282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.001256    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.001931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.001905    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.002358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.002337    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.002736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.002669    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.004826 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.004790    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.005624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.005605    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.059139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.059052    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.059501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.059476    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jnw\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw\") pod \"e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") " pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.121896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.121859    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:41.155927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.155794    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:41.293868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.293831    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188ff58b-dfa2-4a64-9135-616e23d35641" path="/var/lib/kubelet/pods/188ff58b-dfa2-4a64-9135-616e23d35641/volumes"
Oct 30 22:38:41.582154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.582122    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"]
Oct 30 22:38:41.587390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.586565    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:38:41.587787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.587428    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:38:41.587787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.587475    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:38:41.989079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.987830    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" event={"ID":"8791da6c-589d-4f64-b334-bc44be58efb7","Type":"ContainerStarted","Data":"06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"}
Oct 30 22:38:41.989079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:41.987901    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" event={"ID":"8791da6c-589d-4f64-b334-bc44be58efb7","Type":"ContainerStarted","Data":"d5b740a7331404d32663e9852521acd5ae697c18aaebd306165dcaf96e91205d"}
Oct 30 22:38:42.113235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:42.113180    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podStartSLOduration=2.113163109 podStartE2EDuration="2.113163109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:42.10997784 +0000 UTC m=+2252.959490814" watchObservedRunningTime="2026-10-30 22:38:42.113163109 +0000 UTC m=+2252.962676086"
Oct 30 22:38:42.122063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:42.122027    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:42.123975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:42.123945    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:38:42.124114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:42.124004    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:38:43.128495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:43.128454    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:44.010790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:44.010757    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:44.011769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:44.011751    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:38:44.568956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:44.568916    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:44.918582 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:44.918495    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:45.082258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:45.082227    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:45.132840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:45.132795    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerName="payload-processing" containerID="cri-o://f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5" gracePeriod=25
Oct 30 22:38:45.246664 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:45.246622    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerName="payload-pre-processing" containerID="cri-o://007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b" gracePeriod=25
Oct 30 22:38:45.326106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:45.326060    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:46.985467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:46.985427    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:38:46.985910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:46.985436    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:38:48.566935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:48.566894    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:48.762988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:48.762953    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:49.731006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:49.730960    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:49.953759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:49.953722    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:38:49.953993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:49.953978    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:38:54.171568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:54.171535    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:55.666098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:55.666060    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:55.831814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:55.831775    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:56.001252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.001197    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:56.159927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.159894    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:56.220094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.220056    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"]
Oct 30 22:38:56.265295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.265198    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5313f-nf66h"]
Oct 30 22:38:56.640373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.640287    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:56.985045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.985001    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:38:56.985045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:56.985042    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:38:57.294158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:38:57.294079    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93c866f-db20-4126-a3f2-a0c6f1e4e8a8" path="/var/lib/kubelet/pods/b93c866f-db20-4126-a3f2-a0c6f1e4e8a8/volumes"
Oct 30 22:38:58.478093 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:38:58.478050    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:39:00.734461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:00.734394    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:02.868290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:02.868248    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:03.018615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:03.018581    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:03.215273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:03.215244    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:03.781716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:03.781644    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:04.285663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.285632    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"]
Oct 30 22:39:04.294506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.294479    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.336647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.336576    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk97c\" (UniqueName: \"kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.336892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.336755    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.336892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.336842    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.350865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.350832    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"]
Oct 30 22:39:04.389443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.389408    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:04.389656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.389442    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:04.389656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.389442    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:39:04.391776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.391755    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:04.438916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.438879    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk97c\" (UniqueName: \"kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.439114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.439030    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.439192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.439119    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.439891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.439868    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.441796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.441775    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.497659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.497624    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk97c\" (UniqueName: \"kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c\") pod \"maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.610756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.610622    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:04.986153 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:39:04.986104    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2efc9b_59a4_409f_9e2b_7b9be4c72e70.slice/crio-d4d7d12ffe0271a7e79beda61ddd7b6d44c55c7f2f02663994fc8c41e504876c WatchSource:0}: Error finding container d4d7d12ffe0271a7e79beda61ddd7b6d44c55c7f2f02663994fc8c41e504876c: Status 404 returned error can't find the container with id d4d7d12ffe0271a7e79beda61ddd7b6d44c55c7f2f02663994fc8c41e504876c
Oct 30 22:39:04.987315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:04.987286    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"]
Oct 30 22:39:05.049151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.049118    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"]
Oct 30 22:39:05.053298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.053268    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034ca8d2-4hq9p"]
Oct 30 22:39:05.252316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.252237    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz" event={"ID":"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70","Type":"ContainerStarted","Data":"d4d7d12ffe0271a7e79beda61ddd7b6d44c55c7f2f02663994fc8c41e504876c"}
Oct 30 22:39:05.303853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.303815    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34db0655-ba58-4f41-ace7-73e550610cb2" path="/var/lib/kubelet/pods/34db0655-ba58-4f41-ace7-73e550610cb2/volumes"
Oct 30 22:39:05.418738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.418002    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:39:05.454440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.453246    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn\" (UniqueName: \"kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn\") pod \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") "
Oct 30 22:39:05.454440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.453397    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume\") pod \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\" (UID: \"8c6e7f7f-4df8-4202-90c6-211da7ad2951\") "
Oct 30 22:39:05.454885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.454842    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume" pod "8c6e7f7f-4df8-4202-90c6-211da7ad2951" (UID: "8c6e7f7f-4df8-4202-90c6-211da7ad2951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:05.461985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.461940    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn" pod "8c6e7f7f-4df8-4202-90c6-211da7ad2951" (UID: "8c6e7f7f-4df8-4202-90c6-211da7ad2951"). InnerVolumeSpecName "kube-api-access-kz2zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:05.547263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.547129    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:05.556578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.556514    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7f7f-4df8-4202-90c6-211da7ad2951-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:05.556578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.556549    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2zn\" (UniqueName: \"kubernetes.io/projected/8c6e7f7f-4df8-4202-90c6-211da7ad2951-kube-api-access-kz2zn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:05.615399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.615372    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:39:05.657696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.657658    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw\" (UniqueName: \"kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw\") pod \"58798e6e-4fa9-46e7-8c7c-47aec100691b\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") "
Oct 30 22:39:05.657873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.657759    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume\") pod \"58798e6e-4fa9-46e7-8c7c-47aec100691b\" (UID: \"58798e6e-4fa9-46e7-8c7c-47aec100691b\") "
Oct 30 22:39:05.658151 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.658128    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume" pod "58798e6e-4fa9-46e7-8c7c-47aec100691b" (UID: "58798e6e-4fa9-46e7-8c7c-47aec100691b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:05.658335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.658318    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58798e6e-4fa9-46e7-8c7c-47aec100691b-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:05.660020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.659985    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw" pod "58798e6e-4fa9-46e7-8c7c-47aec100691b" (UID: "58798e6e-4fa9-46e7-8c7c-47aec100691b"). InnerVolumeSpecName "kube-api-access-qdxsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:05.760100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:05.760067    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxsw\" (UniqueName: \"kubernetes.io/projected/58798e6e-4fa9-46e7-8c7c-47aec100691b-kube-api-access-qdxsw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:06.261767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.261731    5028 generic.go:356] "Generic (PLEG): container finished" podID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerID="007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b" exitCode=0
Oct 30 22:39:06.261990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.261788    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"
Oct 30 22:39:06.261990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.261815    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" event={"ID":"58798e6e-4fa9-46e7-8c7c-47aec100691b","Type":"ContainerDied","Data":"007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b"}
Oct 30 22:39:06.261990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.261870    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k" event={"ID":"58798e6e-4fa9-46e7-8c7c-47aec100691b","Type":"ContainerDied","Data":"750345500b2616e84cde7e63a2435ef7f205dfb23880d24e93f0158fc7862b11"}
Oct 30 22:39:06.261990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.261892    5028 scope.go:122] "RemoveContainer" containerID="007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b"
Oct 30 22:39:06.263952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.263924    5028 generic.go:356] "Generic (PLEG): container finished" podID="9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" containerID="3933ac9e7bd530a565f15a1f3ee6ba55e511b964494c2ae3113eea1fbcce629b" exitCode=0
Oct 30 22:39:06.264128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.264002    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz" event={"ID":"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70","Type":"ContainerDied","Data":"3933ac9e7bd530a565f15a1f3ee6ba55e511b964494c2ae3113eea1fbcce629b"}
Oct 30 22:39:06.266078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.265775    5028 generic.go:356] "Generic (PLEG): container finished" podID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerID="f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5" exitCode=0
Oct 30 22:39:06.266078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.266053    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" event={"ID":"8c6e7f7f-4df8-4202-90c6-211da7ad2951","Type":"ContainerDied","Data":"f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5"}
Oct 30 22:39:06.266263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.266092    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d" event={"ID":"8c6e7f7f-4df8-4202-90c6-211da7ad2951","Type":"ContainerDied","Data":"ee85279469d612a2bf96fbc3c0606ffa98924560645a17d2beeaef7bb76a35b1"}
Oct 30 22:39:06.266263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.266158    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"
Oct 30 22:39:06.281301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.281275    5028 scope.go:122] "RemoveContainer" containerID="007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b"
Oct 30 22:39:06.281773 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:39:06.281741    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b\": container with ID starting with 007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b not found: ID does not exist" containerID="007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b"
Oct 30 22:39:06.281884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.281780    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b"} err="failed to get container status \"007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b\": rpc error: code = NotFound desc = could not find container \"007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b\": container with ID starting with 007e12191ff589924179d9563b33f36779f7ff9d00148f826c8cc630f6a3f95b not found: ID does not exist"
Oct 30 22:39:06.281884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.281799    5028 scope.go:122] "RemoveContainer" containerID="f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5"
Oct 30 22:39:06.292483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.292463    5028 scope.go:122] "RemoveContainer" containerID="f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5"
Oct 30 22:39:06.292810 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:39:06.292787    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5\": container with ID starting with f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5 not found: ID does not exist" containerID="f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5"
Oct 30 22:39:06.292889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.292817    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5"} err="failed to get container status \"f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5\": rpc error: code = NotFound desc = could not find container \"f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5\": container with ID starting with f5f4432e6ffec7a052b1e10fa4b151daff4db0d8e4d23cb34f91998a512472c5 not found: ID does not exist"
Oct 30 22:39:06.424500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.424422    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"]
Oct 30 22:39:06.469483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.469448    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f80a0e-5bdbb549bf-jq24d"]
Oct 30 22:39:06.533491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.533450    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"]
Oct 30 22:39:06.540214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:06.540181    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f80a0e-7fbf86d675-7m46k"]
Oct 30 22:39:07.293083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.293053    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" path="/var/lib/kubelet/pods/58798e6e-4fa9-46e7-8c7c-47aec100691b/volumes"
Oct 30 22:39:07.293512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.293498    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" path="/var/lib/kubelet/pods/8c6e7f7f-4df8-4202-90c6-211da7ad2951/volumes"
Oct 30 22:39:07.446438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.446410    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:07.481670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.481635    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c\" (UniqueName: \"kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c\") pod \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") "
Oct 30 22:39:07.481854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.481798    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca\") pod \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") "
Oct 30 22:39:07.481944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.481929    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca\") pod \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\" (UID: \"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70\") "
Oct 30 22:39:07.482201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.482172    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca" pod "9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" (UID: "9f2efc9b-59a4-409f-9e2b-7b9be4c72e70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:07.482464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.482446    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:07.483789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.483764    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c" pod "9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" (UID: "9f2efc9b-59a4-409f-9e2b-7b9be4c72e70"). InnerVolumeSpecName "kube-api-access-fk97c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:07.483961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.483943    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca" pod "9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" (UID: "9f2efc9b-59a4-409f-9e2b-7b9be4c72e70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:07.584360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.584278    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk97c\" (UniqueName: \"kubernetes.io/projected/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-kube-api-access-fk97c\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:07.584360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:07.584310    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:08.287893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:08.287860    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"
Oct 30 22:39:08.288082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:08.287866    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz" event={"ID":"9f2efc9b-59a4-409f-9e2b-7b9be4c72e70","Type":"ContainerDied","Data":"d4d7d12ffe0271a7e79beda61ddd7b6d44c55c7f2f02663994fc8c41e504876c"}
Oct 30 22:39:08.288082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:08.287982    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d7d12ffe0271a7e79beda61ddd7b6d44c55c7f2f02663994fc8c41e504876c"
Oct 30 22:39:08.906524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:08.906494    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:09.570163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:09.570121    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:09.719122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:09.719088    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:09.874126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:09.874038    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:10.437612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:10.437578    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:12.849257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:12.849220    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:14.615968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.615927    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"]
Oct 30 22:39:14.616986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.616961    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" containerName="revoke-keys"
Oct 30 22:39:14.616986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.616985    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" containerName="revoke-keys"
Oct 30 22:39:14.617180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617095    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerName="payload-processing"
Oct 30 22:39:14.617180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617104    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerName="payload-processing"
Oct 30 22:39:14.617180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617148    5028 cpu_manager.go:414] "containerMap: removing container" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerName="payload-pre-processing"
Oct 30 22:39:14.617180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617157    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerName="payload-pre-processing"
Oct 30 22:39:14.617389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617352    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6e7f7f-4df8-4202-90c6-211da7ad2951" containerName="payload-processing"
Oct 30 22:39:14.617389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617368    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="58798e6e-4fa9-46e7-8c7c-47aec100691b" containerName="payload-pre-processing"
Oct 30 22:39:14.617389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.617381    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" containerName="revoke-keys"
Oct 30 22:39:14.622447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.622425    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.664194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.664157    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"]
Oct 30 22:39:14.669027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.669003    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.675863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.675832    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"]
Oct 30 22:39:14.695591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.695549    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.695809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.695783    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7kp\" (UniqueName: \"kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.695927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.695906    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.706852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.706821    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:14.707006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.706857    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:14.716811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.716764    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:14.716811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.716786    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:39:14.724583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.724554    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"]
Oct 30 22:39:14.758533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.758502    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"]
Oct 30 22:39:14.797779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.797737    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6l7l\" (UniqueName: \"kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.797992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.797794    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7kp\" (UniqueName: \"kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.797992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.797896    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.798123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.798028    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.798179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.798136    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.798237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.798178    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.798799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.798771    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.800451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.800429    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.834207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.834169    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"]
Oct 30 22:39:14.856544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.856510    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7kp\" (UniqueName: \"kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp\") pod \"maas-api-revoke-sub-61fcc509-2d47p\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.899382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.899290    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6l7l\" (UniqueName: \"kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.899539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.899473    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.899610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.899586    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.900267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.900241    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.901870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.901843    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.937517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.937488    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:14.958239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.958207    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6l7l\" (UniqueName: \"kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l\") pod \"maas-api-revoke-sub-648961eb-sb9j8\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:14.988876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:14.988840    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:15.090729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:15.090645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:15.239114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:15.239075    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:15.325427 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:39:15.325392    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b67eee3_3b75_4237_b2f7_0261fff98e2d.slice/crio-f220f69d233bef82ead0a92d6a27797160a84027ac6d4bbf37de56bf79139443 WatchSource:0}: Error finding container f220f69d233bef82ead0a92d6a27797160a84027ac6d4bbf37de56bf79139443: Status 404 returned error can't find the container with id f220f69d233bef82ead0a92d6a27797160a84027ac6d4bbf37de56bf79139443
Oct 30 22:39:15.326727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:15.326700    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"]
Oct 30 22:39:15.402410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:15.402346    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p" event={"ID":"2b67eee3-3b75-4237-b2f7-0261fff98e2d","Type":"ContainerStarted","Data":"f220f69d233bef82ead0a92d6a27797160a84027ac6d4bbf37de56bf79139443"}
Oct 30 22:39:15.413922 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:39:15.413893    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbbb719_03b2_4c1c_828f_4373fe8996c7.slice/crio-82ee39b23ed6829f11740f6badc1a20ab786e063b73e1606b3bedc76940a7c88 WatchSource:0}: Error finding container 82ee39b23ed6829f11740f6badc1a20ab786e063b73e1606b3bedc76940a7c88: Status 404 returned error can't find the container with id 82ee39b23ed6829f11740f6badc1a20ab786e063b73e1606b3bedc76940a7c88
Oct 30 22:39:15.414120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:15.414094    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"]
Oct 30 22:39:15.574741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:15.574708    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:16.413927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:16.413887    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8" event={"ID":"8dbbb719-03b2-4c1c-828f-4373fe8996c7","Type":"ContainerStarted","Data":"fbb13c2d04ce8758c557e7233045b62093f1741c0c42afd77c73dd22c0512464"}
Oct 30 22:39:16.414390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:16.413949    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8" event={"ID":"8dbbb719-03b2-4c1c-828f-4373fe8996c7","Type":"ContainerStarted","Data":"82ee39b23ed6829f11740f6badc1a20ab786e063b73e1606b3bedc76940a7c88"}
Oct 30 22:39:16.415949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:16.415913    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p" event={"ID":"2b67eee3-3b75-4237-b2f7-0261fff98e2d","Type":"ContainerStarted","Data":"0b4aa04809ef35e1a034b2f7683855cf37714bc244df567073439bc429649f83"}
Oct 30 22:39:16.546383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:16.546301    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8" podStartSLOduration=2.5462841000000003 podStartE2EDuration="2.5462841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:16.531222333 +0000 UTC m=+2287.380735306" watchObservedRunningTime="2026-10-30 22:39:16.5462841 +0000 UTC m=+2287.395797072"
Oct 30 22:39:16.645915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:16.645862    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p" podStartSLOduration=2.645845314 podStartE2EDuration="2.645845314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:16.642831342 +0000 UTC m=+2287.492344316" watchObservedRunningTime="2026-10-30 22:39:16.645845314 +0000 UTC m=+2287.495358284"
Oct 30 22:39:16.954600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:16.954563    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:17.122016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.121971    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:17.426751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.426711    5028 generic.go:356] "Generic (PLEG): container finished" podID="2b67eee3-3b75-4237-b2f7-0261fff98e2d" containerID="0b4aa04809ef35e1a034b2f7683855cf37714bc244df567073439bc429649f83" exitCode=0
Oct 30 22:39:17.427224 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.426786    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p" event={"ID":"2b67eee3-3b75-4237-b2f7-0261fff98e2d","Type":"ContainerDied","Data":"0b4aa04809ef35e1a034b2f7683855cf37714bc244df567073439bc429649f83"}
Oct 30 22:39:17.428620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.428595    5028 generic.go:356] "Generic (PLEG): container finished" podID="8dbbb719-03b2-4c1c-828f-4373fe8996c7" containerID="fbb13c2d04ce8758c557e7233045b62093f1741c0c42afd77c73dd22c0512464" exitCode=0
Oct 30 22:39:17.428785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.428673    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8" event={"ID":"8dbbb719-03b2-4c1c-828f-4373fe8996c7","Type":"ContainerDied","Data":"fbb13c2d04ce8758c557e7233045b62093f1741c0c42afd77c73dd22c0512464"}
Oct 30 22:39:17.668552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.668518    5028 scope.go:122] "RemoveContainer" containerID="eaf5f583b22920bdd6bfd8e316928c3956dae7d27cb3e26bebe8bbd0573c28bd"
Oct 30 22:39:17.681659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:17.681632    5028 scope.go:122] "RemoveContainer" containerID="fbc6bbebc09c2effc64e0a65b0d2637e11bbcc3d14cc2d06883cfc2821b8618b"
Oct 30 22:39:18.652309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.651888    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:18.697223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.697155    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:18.788785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.788744    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca\") pod \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") "
Oct 30 22:39:18.788988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.788847    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l\" (UniqueName: \"kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l\") pod \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") "
Oct 30 22:39:18.789055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789018    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca\") pod \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") "
Oct 30 22:39:18.789104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789076    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca\") pod \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") "
Oct 30 22:39:18.789199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789150    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca" pod "2b67eee3-3b75-4237-b2f7-0261fff98e2d" (UID: "2b67eee3-3b75-4237-b2f7-0261fff98e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:18.789199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789184    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca\") pod \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\" (UID: \"8dbbb719-03b2-4c1c-828f-4373fe8996c7\") "
Oct 30 22:39:18.789314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789239    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp\" (UniqueName: \"kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp\") pod \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\" (UID: \"2b67eee3-3b75-4237-b2f7-0261fff98e2d\") "
Oct 30 22:39:18.789466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789437    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca" pod "8dbbb719-03b2-4c1c-828f-4373fe8996c7" (UID: "8dbbb719-03b2-4c1c-828f-4373fe8996c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:18.789983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789958    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:18.790103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.789987    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:18.792068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.792039    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca" pod "2b67eee3-3b75-4237-b2f7-0261fff98e2d" (UID: "2b67eee3-3b75-4237-b2f7-0261fff98e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:18.792068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.792046    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca" pod "8dbbb719-03b2-4c1c-828f-4373fe8996c7" (UID: "8dbbb719-03b2-4c1c-828f-4373fe8996c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:18.792252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.792069    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l" pod "8dbbb719-03b2-4c1c-828f-4373fe8996c7" (UID: "8dbbb719-03b2-4c1c-828f-4373fe8996c7"). InnerVolumeSpecName "kube-api-access-t6l7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:18.794019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.793989    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp" pod "2b67eee3-3b75-4237-b2f7-0261fff98e2d" (UID: "2b67eee3-3b75-4237-b2f7-0261fff98e2d"). InnerVolumeSpecName "kube-api-access-nn7kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:18.891526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.891489    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b67eee3-3b75-4237-b2f7-0261fff98e2d-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:18.891526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.891519    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbbb719-03b2-4c1c-828f-4373fe8996c7-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:18.891526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.891531    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn7kp\" (UniqueName: \"kubernetes.io/projected/2b67eee3-3b75-4237-b2f7-0261fff98e2d-kube-api-access-nn7kp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:18.891800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:18.891540    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6l7l\" (UniqueName: \"kubernetes.io/projected/8dbbb719-03b2-4c1c-828f-4373fe8996c7-kube-api-access-t6l7l\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:19.450451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.450415    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"
Oct 30 22:39:19.450657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.450448    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p" event={"ID":"2b67eee3-3b75-4237-b2f7-0261fff98e2d","Type":"ContainerDied","Data":"f220f69d233bef82ead0a92d6a27797160a84027ac6d4bbf37de56bf79139443"}
Oct 30 22:39:19.450657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.450508    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f220f69d233bef82ead0a92d6a27797160a84027ac6d4bbf37de56bf79139443"
Oct 30 22:39:19.452291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.452262    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8" event={"ID":"8dbbb719-03b2-4c1c-828f-4373fe8996c7","Type":"ContainerDied","Data":"82ee39b23ed6829f11740f6badc1a20ab786e063b73e1606b3bedc76940a7c88"}
Oct 30 22:39:19.452406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.452312    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ee39b23ed6829f11740f6badc1a20ab786e063b73e1606b3bedc76940a7c88"
Oct 30 22:39:19.452406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.452277    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"
Oct 30 22:39:19.759163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.759121    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerName="payload-processing" containerID="cri-o://a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0" gracePeriod=25
Oct 30 22:39:19.835295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:19.835247    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerName="payload-pre-processing" containerID="cri-o://8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd" gracePeriod=25
Oct 30 22:39:20.389753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:20.389722    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:21.951478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:21.951436    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:21.951942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:21.951518    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:22.461426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:22.461392    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:22.746767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:22.744602    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.066249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:25.066205    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:28.453942 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:39:28.453895    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:39:29.663732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:29.663671    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:29.824717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:29.817164    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:31.951666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:31.951629    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:31.951666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:31.951646    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:34.130940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:34.130901    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:34.297774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:34.297737    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:34.687659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:34.687571    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:37.248897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:37.248854    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:39:37.249319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:37.248923    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:39:37.883429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:37.883397    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:38.033450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.031517    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:38.751666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.751634    5028 generic.go:356] "Generic (PLEG): container finished" podID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerID="89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a" exitCode=0
Oct 30 22:39:38.752162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.751735    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" event={"ID":"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de","Type":"ContainerDied","Data":"89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"}
Oct 30 22:39:38.752270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.752252    5028 scope.go:122] "RemoveContainer" containerID="89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"
Oct 30 22:39:38.754306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.754270    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:39:38.754445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.754363    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:39:38.754445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.754408    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:39:38.793578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:38.793546    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:39.762787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:39.762746    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" event={"ID":"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de","Type":"ContainerStarted","Data":"af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf"}
Oct 30 22:39:40.063539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.062982    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:39:40.063539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.063116    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:39:40.063539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.063171    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 22:39:40.115515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.115481    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:39:40.197814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.197790    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:39:40.264159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.264098    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc\" (UniqueName: \"kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc\") pod \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") "
Oct 30 22:39:40.264378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.264266    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume\") pod \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\" (UID: \"3d8a636e-75a3-4da7-8ccf-058f43de40aa\") "
Oct 30 22:39:40.264939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.264891    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume" pod "3d8a636e-75a3-4da7-8ccf-058f43de40aa" (UID: "3d8a636e-75a3-4da7-8ccf-058f43de40aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:40.265400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.265371    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8a636e-75a3-4da7-8ccf-058f43de40aa-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:40.267895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.267858    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc" pod "3d8a636e-75a3-4da7-8ccf-058f43de40aa" (UID: "3d8a636e-75a3-4da7-8ccf-058f43de40aa"). InnerVolumeSpecName "kube-api-access-vflvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:40.366574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.366483    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume\") pod \"2368593a-b34f-4133-b1cd-1e09488a8ca0\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") "
Oct 30 22:39:40.366574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.366533    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t\" (UniqueName: \"kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t\") pod \"2368593a-b34f-4133-b1cd-1e09488a8ca0\" (UID: \"2368593a-b34f-4133-b1cd-1e09488a8ca0\") "
Oct 30 22:39:40.369730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.367235    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vflvc\" (UniqueName: \"kubernetes.io/projected/3d8a636e-75a3-4da7-8ccf-058f43de40aa-kube-api-access-vflvc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:40.370368 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.370341    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume" pod "2368593a-b34f-4133-b1cd-1e09488a8ca0" (UID: "2368593a-b34f-4133-b1cd-1e09488a8ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:40.373359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.373333    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t" pod "2368593a-b34f-4133-b1cd-1e09488a8ca0" (UID: "2368593a-b34f-4133-b1cd-1e09488a8ca0"). InnerVolumeSpecName "kube-api-access-4s82t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:40.468936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.468902    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2368593a-b34f-4133-b1cd-1e09488a8ca0-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:40.468936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.468936    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s82t\" (UniqueName: \"kubernetes.io/projected/2368593a-b34f-4133-b1cd-1e09488a8ca0-kube-api-access-4s82t\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:40.775571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.775534    5028 generic.go:356] "Generic (PLEG): container finished" podID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerID="8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd" exitCode=0
Oct 30 22:39:40.776057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.775579    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"
Oct 30 22:39:40.776057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.775609    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" event={"ID":"2368593a-b34f-4133-b1cd-1e09488a8ca0","Type":"ContainerDied","Data":"8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd"}
Oct 30 22:39:40.776057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.775667    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk" event={"ID":"2368593a-b34f-4133-b1cd-1e09488a8ca0","Type":"ContainerDied","Data":"40833ef72bf7eb9ff28af42fc38409f0734d06e0bf117a83fe0d44013a491d2d"}
Oct 30 22:39:40.776057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.775714    5028 scope.go:122] "RemoveContainer" containerID="8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd"
Oct 30 22:39:40.777669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.777644    5028 generic.go:356] "Generic (PLEG): container finished" podID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerID="a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0" exitCode=0
Oct 30 22:39:40.778560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.778540    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"
Oct 30 22:39:40.780980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.780814    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" event={"ID":"3d8a636e-75a3-4da7-8ccf-058f43de40aa","Type":"ContainerDied","Data":"a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0"}
Oct 30 22:39:40.780980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.780869    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn" event={"ID":"3d8a636e-75a3-4da7-8ccf-058f43de40aa","Type":"ContainerDied","Data":"49047c76cd06ec5fc4280ba58fd982cdc5c324fed798540d7f178e4ae4ddc07e"}
Oct 30 22:39:40.790299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.790277    5028 scope.go:122] "RemoveContainer" containerID="8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd"
Oct 30 22:39:40.790623 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:39:40.790603    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd\": container with ID starting with 8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd not found: ID does not exist" containerID="8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd"
Oct 30 22:39:40.790671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.790633    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd"} err="failed to get container status \"8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd\": rpc error: code = NotFound desc = could not find container \"8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd\": container with ID starting with 8fdb91c8e25cc322d45bb7e53bdd5a65c9f4d87b5eae77eee9132be4a424abbd not found: ID does not exist"
Oct 30 22:39:40.790671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.790652    5028 scope.go:122] "RemoveContainer" containerID="a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0"
Oct 30 22:39:40.801157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.801131    5028 scope.go:122] "RemoveContainer" containerID="a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0"
Oct 30 22:39:40.801527 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:39:40.801497    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0\": container with ID starting with a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0 not found: ID does not exist" containerID="a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0"
Oct 30 22:39:40.801632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.801539    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0"} err="failed to get container status \"a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0\": rpc error: code = NotFound desc = could not find container \"a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0\": container with ID starting with a19d56a8b6f60e664ecd8408059eebc42a475002f3c0b498de25140378cd0ce0 not found: ID does not exist"
Oct 30 22:39:40.961504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.961453    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"]
Oct 30 22:39:40.968284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:40.968252    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-af30a9-b44c8fc5c-7jxcn"]
Oct 30 22:39:41.056438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:41.056348    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:39:41.056438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:41.056411    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 22:39:41.069569 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:41.069531    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"]
Oct 30 22:39:41.077312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:41.076752    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-af30a9-78db456cd7-7k9hk"]
Oct 30 22:39:41.293828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:41.293787    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" path="/var/lib/kubelet/pods/2368593a-b34f-4133-b1cd-1e09488a8ca0/volumes"
Oct 30 22:39:41.294254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:41.294239    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" path="/var/lib/kubelet/pods/3d8a636e-75a3-4da7-8ccf-058f43de40aa/volumes"
Oct 30 22:39:42.056103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.056057    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:39:42.056603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.056132    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 22:39:42.777732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.777681    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"]
Oct 30 22:39:42.778407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778376    5028 cpu_manager.go:414] "containerMap: removing container" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerName="payload-processing"
Oct 30 22:39:42.778407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778391    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerName="payload-processing"
Oct 30 22:39:42.778499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778424    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerName="payload-pre-processing"
Oct 30 22:39:42.778499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778430    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerName="payload-pre-processing"
Oct 30 22:39:42.778499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778443    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8dbbb719-03b2-4c1c-828f-4373fe8996c7" containerName="revoke-keys"
Oct 30 22:39:42.778499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778448    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbbb719-03b2-4c1c-828f-4373fe8996c7" containerName="revoke-keys"
Oct 30 22:39:42.778499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778456    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2b67eee3-3b75-4237-b2f7-0261fff98e2d" containerName="revoke-keys"
Oct 30 22:39:42.778499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778461    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b67eee3-3b75-4237-b2f7-0261fff98e2d" containerName="revoke-keys"
Oct 30 22:39:42.778734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778575    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8a636e-75a3-4da7-8ccf-058f43de40aa" containerName="payload-processing"
Oct 30 22:39:42.778734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778589    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2368593a-b34f-4133-b1cd-1e09488a8ca0" containerName="payload-pre-processing"
Oct 30 22:39:42.778734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778595    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbbb719-03b2-4c1c-828f-4373fe8996c7" containerName="revoke-keys"
Oct 30 22:39:42.778734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.778601    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b67eee3-3b75-4237-b2f7-0261fff98e2d" containerName="revoke-keys"
Oct 30 22:39:42.782105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.782086    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.800564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.800529    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.800762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.800739    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnzn\" (UniqueName: \"kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.800831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.800788    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.855402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.855364    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"]
Oct 30 22:39:42.862138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.862107    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"]
Oct 30 22:39:42.862402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.862240    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:42.874148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.872179    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:42.888597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.888566    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:42.888786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.888718    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:39:42.888786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.888760    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:42.896166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.896135    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"]
Oct 30 22:39:42.901938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.901912    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.902066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.901983    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:42.902126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.902099    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbqc\" (UniqueName: \"kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:42.902200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.902183    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:42.902285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.902270    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnzn\" (UniqueName: \"kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.902341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.902328    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.903071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.903046    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.905200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.905180    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:42.981600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:42.981480    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnzn\" (UniqueName: \"kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn\") pod \"maas-api-revoke-sub-96d7d79d-8zlj9\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:43.003383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.003351    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.003570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.003430    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbqc\" (UniqueName: \"kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.003570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.003492    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.004202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.004176    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.006107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.006084    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.055909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.055813    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:39:43.055909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.055868    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 22:39:43.062861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.062829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbqc\" (UniqueName: \"kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc\") pod \"maas-api-revoke-sub-240bacd2-bqhs4\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.095846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.095813    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:43.180254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.180219    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:43.496282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.493662    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:43.512579 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:39:43.512544    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac38bc2_6733_4594_831a_e07764f2e3ce.slice/crio-6943df52c0e34c0878bdb609686400a45b8cf1f4ac7fa2c2e0546bba4a392234 WatchSource:0}: Error finding container 6943df52c0e34c0878bdb609686400a45b8cf1f4ac7fa2c2e0546bba4a392234: Status 404 returned error can't find the container with id 6943df52c0e34c0878bdb609686400a45b8cf1f4ac7fa2c2e0546bba4a392234
Oct 30 22:39:43.512716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.512593    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"]
Oct 30 22:39:43.664222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.663751    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:43.697860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.697826    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"]
Oct 30 22:39:43.807190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.807153    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9" event={"ID":"8ac38bc2-6733-4594-831a-e07764f2e3ce","Type":"ContainerStarted","Data":"6943df52c0e34c0878bdb609686400a45b8cf1f4ac7fa2c2e0546bba4a392234"}
Oct 30 22:39:43.808420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:43.808395    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4" event={"ID":"57dfad38-3130-45a1-8ad7-5147d62da656","Type":"ContainerStarted","Data":"87ae18e431b399dc681ef4fa047ba1a2fee76ecd1cda5b7f697eb60e8c73a626"}
Oct 30 22:39:44.024717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.022734    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:44.055825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.055786    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:39:44.056055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.055840    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 22:39:44.491660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.491627    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:44.856903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.856862    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9" event={"ID":"8ac38bc2-6733-4594-831a-e07764f2e3ce","Type":"ContainerStarted","Data":"8227324ed15951408b501e879dddc7f4566b204c08882ebfb7bfe87dfa0f45f5"}
Oct 30 22:39:44.858979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.858946    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4" event={"ID":"57dfad38-3130-45a1-8ad7-5147d62da656","Type":"ContainerStarted","Data":"7b5eba30fca24c9d248cca49cf99bb5eb34756451966df418188395cde61e961"}
Oct 30 22:39:44.975071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:44.975007    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9" podStartSLOduration=2.974990062 podStartE2EDuration="2.974990062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:44.973406214 +0000 UTC m=+2315.822919189" watchObservedRunningTime="2026-10-30 22:39:44.974990062 +0000 UTC m=+2315.824503034"
Oct 30 22:39:45.003820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.003784    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"]
Oct 30 22:39:45.007394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.007373    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.055445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.055335    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:39:45.055445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.055414    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 22:39:45.066251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.066213    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrwt\" (UniqueName: \"kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.066448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.066415    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.066521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.066470    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.104741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.090552    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"]
Oct 30 22:39:45.104741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.090806    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4" podStartSLOduration=3.090789406 podStartE2EDuration="3.090789406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:45.085300544 +0000 UTC m=+2315.934813518" watchObservedRunningTime="2026-10-30 22:39:45.090789406 +0000 UTC m=+2315.940302380"
Oct 30 22:39:45.168615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.168515    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrwt\" (UniqueName: \"kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.168827 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.168712    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.169594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.169387    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.170223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.170203    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.173760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.173731    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.228478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.228447    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrwt\" (UniqueName: \"kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt\") pod \"maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.403698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.403660    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:45.822007 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:39:45.821972    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceed8001_d99a_443e_b0de_1fb6c094b5e7.slice/crio-3a2754b69fdc2e0996cbfeba703b791628ad913a7c7778c4014dd3eedd217e21 WatchSource:0}: Error finding container 3a2754b69fdc2e0996cbfeba703b791628ad913a7c7778c4014dd3eedd217e21: Status 404 returned error can't find the container with id 3a2754b69fdc2e0996cbfeba703b791628ad913a7c7778c4014dd3eedd217e21
Oct 30 22:39:45.825485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.825451    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"]
Oct 30 22:39:45.876908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.876874    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5" event={"ID":"ceed8001-d99a-443e-b0de-1fb6c094b5e7","Type":"ContainerStarted","Data":"3a2754b69fdc2e0996cbfeba703b791628ad913a7c7778c4014dd3eedd217e21"}
Oct 30 22:39:45.878998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.878969    5028 generic.go:356] "Generic (PLEG): container finished" podID="8ac38bc2-6733-4594-831a-e07764f2e3ce" containerID="8227324ed15951408b501e879dddc7f4566b204c08882ebfb7bfe87dfa0f45f5" exitCode=0
Oct 30 22:39:45.879114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.879049    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9" event={"ID":"8ac38bc2-6733-4594-831a-e07764f2e3ce","Type":"ContainerDied","Data":"8227324ed15951408b501e879dddc7f4566b204c08882ebfb7bfe87dfa0f45f5"}
Oct 30 22:39:45.881078 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.881056    5028 generic.go:356] "Generic (PLEG): container finished" podID="57dfad38-3130-45a1-8ad7-5147d62da656" containerID="7b5eba30fca24c9d248cca49cf99bb5eb34756451966df418188395cde61e961" exitCode=0
Oct 30 22:39:45.881183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:45.881136    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4" event={"ID":"57dfad38-3130-45a1-8ad7-5147d62da656","Type":"ContainerDied","Data":"7b5eba30fca24c9d248cca49cf99bb5eb34756451966df418188395cde61e961"}
Oct 30 22:39:46.060796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:46.060763    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:39:46.061182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:46.061154    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:39:46.062024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:46.062006    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:39:46.890778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:46.890746    5028 generic.go:356] "Generic (PLEG): container finished" podID="ceed8001-d99a-443e-b0de-1fb6c094b5e7" containerID="622d1ef67a4afd7b4d4aa7ea9b238ee81d3db9fb50ca72e6897488d9c0f00a2f" exitCode=0
Oct 30 22:39:46.891220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:46.890838    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5" event={"ID":"ceed8001-d99a-443e-b0de-1fb6c094b5e7","Type":"ContainerDied","Data":"622d1ef67a4afd7b4d4aa7ea9b238ee81d3db9fb50ca72e6897488d9c0f00a2f"}
Oct 30 22:39:47.107636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.107605    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:47.114468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.114380    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:47.190405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.190372    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.210261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210226    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn\" (UniqueName: \"kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn\") pod \"8ac38bc2-6733-4594-831a-e07764f2e3ce\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") "
Oct 30 22:39:47.210459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210278    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca\") pod \"8ac38bc2-6733-4594-831a-e07764f2e3ce\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") "
Oct 30 22:39:47.210459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210342    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc\" (UniqueName: \"kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc\") pod \"57dfad38-3130-45a1-8ad7-5147d62da656\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") "
Oct 30 22:39:47.210459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210371    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca\") pod \"57dfad38-3130-45a1-8ad7-5147d62da656\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") "
Oct 30 22:39:47.210459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210405    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca\") pod \"57dfad38-3130-45a1-8ad7-5147d62da656\" (UID: \"57dfad38-3130-45a1-8ad7-5147d62da656\") "
Oct 30 22:39:47.210731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210637    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca\") pod \"8ac38bc2-6733-4594-831a-e07764f2e3ce\" (UID: \"8ac38bc2-6733-4594-831a-e07764f2e3ce\") "
Oct 30 22:39:47.210731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.210676    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca" pod "8ac38bc2-6733-4594-831a-e07764f2e3ce" (UID: "8ac38bc2-6733-4594-831a-e07764f2e3ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:47.211097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.211070    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca" pod "57dfad38-3130-45a1-8ad7-5147d62da656" (UID: "57dfad38-3130-45a1-8ad7-5147d62da656"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:47.211483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.211456    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.211581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.211489    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.213896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.213869    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn" pod "8ac38bc2-6733-4594-831a-e07764f2e3ce" (UID: "8ac38bc2-6733-4594-831a-e07764f2e3ce"). InnerVolumeSpecName "kube-api-access-tnnzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:47.214004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.213971    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca" pod "8ac38bc2-6733-4594-831a-e07764f2e3ce" (UID: "8ac38bc2-6733-4594-831a-e07764f2e3ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:47.214119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.214100    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc" pod "57dfad38-3130-45a1-8ad7-5147d62da656" (UID: "57dfad38-3130-45a1-8ad7-5147d62da656"). InnerVolumeSpecName "kube-api-access-4sbqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:47.214185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.214147    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca" pod "57dfad38-3130-45a1-8ad7-5147d62da656" (UID: "57dfad38-3130-45a1-8ad7-5147d62da656"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:47.319320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.319233    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnnzn\" (UniqueName: \"kubernetes.io/projected/8ac38bc2-6733-4594-831a-e07764f2e3ce-kube-api-access-tnnzn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.319320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.319266    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbqc\" (UniqueName: \"kubernetes.io/projected/57dfad38-3130-45a1-8ad7-5147d62da656-kube-api-access-4sbqc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.319320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.319277    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dfad38-3130-45a1-8ad7-5147d62da656-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.319320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.319286    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac38bc2-6733-4594-831a-e07764f2e3ce-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.344062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.344016    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.915794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.915758    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.917643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.917615    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9" event={"ID":"8ac38bc2-6733-4594-831a-e07764f2e3ce","Type":"ContainerDied","Data":"6943df52c0e34c0878bdb609686400a45b8cf1f4ac7fa2c2e0546bba4a392234"}
Oct 30 22:39:47.917868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.917671    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6943df52c0e34c0878bdb609686400a45b8cf1f4ac7fa2c2e0546bba4a392234"
Oct 30 22:39:47.917868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.917763    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"
Oct 30 22:39:47.925136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.925042    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"
Oct 30 22:39:47.925757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.925708    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4" event={"ID":"57dfad38-3130-45a1-8ad7-5147d62da656","Type":"ContainerDied","Data":"87ae18e431b399dc681ef4fa047ba1a2fee76ecd1cda5b7f697eb60e8c73a626"}
Oct 30 22:39:47.926327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:47.925766    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ae18e431b399dc681ef4fa047ba1a2fee76ecd1cda5b7f697eb60e8c73a626"
Oct 30 22:39:48.135426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.135399    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:48.231992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.231954    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca\") pod \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") "
Oct 30 22:39:48.232178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.232046    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt\" (UniqueName: \"kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt\") pod \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") "
Oct 30 22:39:48.232178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.232092    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca\") pod \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\" (UID: \"ceed8001-d99a-443e-b0de-1fb6c094b5e7\") "
Oct 30 22:39:48.232527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.232502    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca" pod "ceed8001-d99a-443e-b0de-1fb6c094b5e7" (UID: "ceed8001-d99a-443e-b0de-1fb6c094b5e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:48.232642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.232628    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.234205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.234178    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca" pod "ceed8001-d99a-443e-b0de-1fb6c094b5e7" (UID: "ceed8001-d99a-443e-b0de-1fb6c094b5e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:48.234314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.234173    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt" pod "ceed8001-d99a-443e-b0de-1fb6c094b5e7" (UID: "ceed8001-d99a-443e-b0de-1fb6c094b5e7"). InnerVolumeSpecName "kube-api-access-qlrwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:48.334553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.334506    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceed8001-d99a-443e-b0de-1fb6c094b5e7-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.334553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.334546    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlrwt\" (UniqueName: \"kubernetes.io/projected/ceed8001-d99a-443e-b0de-1fb6c094b5e7-kube-api-access-qlrwt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.649466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.649378    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:48.937079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.937005    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"
Oct 30 22:39:48.937079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.937036    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5" event={"ID":"ceed8001-d99a-443e-b0de-1fb6c094b5e7","Type":"ContainerDied","Data":"3a2754b69fdc2e0996cbfeba703b791628ad913a7c7778c4014dd3eedd217e21"}
Oct 30 22:39:48.937489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:48.937112    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2754b69fdc2e0996cbfeba703b791628ad913a7c7778c4014dd3eedd217e21"
Oct 30 22:39:49.411485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:49.411446    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"]
Oct 30 22:39:49.433931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:49.433882    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"]
Oct 30 22:39:49.610292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:49.610257    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:53.497787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:53.497747    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:53.732753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:53.732719    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:56.419500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:56.419462    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"]
Oct 30 22:39:56.805386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:56.805357    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:56.971711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:56.969139    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:58.335619 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:39:58.335584    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:58.447636 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:39:58.447591    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:40:01.420215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:01.420160    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerName="payload-processing" containerID="cri-o://3a58f9b69f35ff7b26879846ffed044a67d2acccb1ed0a0a99f3724533d437cd" gracePeriod=25
Oct 30 22:40:03.208713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:03.204736    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:04.433843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.433797    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main" containerID="cri-o://139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754" gracePeriod=45
Oct 30 22:40:04.460210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.460166    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" containerID="cri-o://dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087" gracePeriod=45
Oct 30 22:40:04.646885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.646842    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 22:40:04.799460 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.799431    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:40:04.802951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.802930    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:40:04.813731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.813681    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir\") pod \"669e0382-43ca-47fe-9926-4d9634ce133f\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") "
Oct 30 22:40:04.813817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.813797    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc\" (UniqueName: \"kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc\") pod \"ccd239f8-de79-43a0-af71-bf140440a576\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") "
Oct 30 22:40:04.813873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.813846    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache\") pod \"ccd239f8-de79-43a0-af71-bf140440a576\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") "
Oct 30 22:40:04.813919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.813884    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx\" (UniqueName: \"kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx\") pod \"669e0382-43ca-47fe-9926-4d9634ce133f\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") "
Oct 30 22:40:04.813981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.813924    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir\") pod \"ccd239f8-de79-43a0-af71-bf140440a576\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") "
Oct 30 22:40:04.813981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.813958    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir" pod "669e0382-43ca-47fe-9926-4d9634ce133f" (UID: "669e0382-43ca-47fe-9926-4d9634ce133f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.814101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814013    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home\") pod \"ccd239f8-de79-43a0-af71-bf140440a576\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") "
Oct 30 22:40:04.814101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814054    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home\") pod \"669e0382-43ca-47fe-9926-4d9634ce133f\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") "
Oct 30 22:40:04.814101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814094    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs\") pod \"ccd239f8-de79-43a0-af71-bf140440a576\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") "
Oct 30 22:40:04.814257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814122    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm\") pod \"ccd239f8-de79-43a0-af71-bf140440a576\" (UID: \"ccd239f8-de79-43a0-af71-bf140440a576\") "
Oct 30 22:40:04.814257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814154    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm\") pod \"669e0382-43ca-47fe-9926-4d9634ce133f\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") "
Oct 30 22:40:04.814257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814174    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache" pod "ccd239f8-de79-43a0-af71-bf140440a576" (UID: "ccd239f8-de79-43a0-af71-bf140440a576"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.814257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814218    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs\" (UniqueName: \"kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs\") pod \"669e0382-43ca-47fe-9926-4d9634ce133f\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") "
Oct 30 22:40:04.814257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814238    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache\") pod \"669e0382-43ca-47fe-9926-4d9634ce133f\" (UID: \"669e0382-43ca-47fe-9926-4d9634ce133f\") "
Oct 30 22:40:04.814535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814214    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir" pod "ccd239f8-de79-43a0-af71-bf140440a576" (UID: "ccd239f8-de79-43a0-af71-bf140440a576"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.814535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814416    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home" pod "669e0382-43ca-47fe-9926-4d9634ce133f" (UID: "669e0382-43ca-47fe-9926-4d9634ce133f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.814661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814581    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home" pod "ccd239f8-de79-43a0-af71-bf140440a576" (UID: "ccd239f8-de79-43a0-af71-bf140440a576"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.814840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.814815    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache" pod "669e0382-43ca-47fe-9926-4d9634ce133f" (UID: "669e0382-43ca-47fe-9926-4d9634ce133f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.815090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.815067    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.815179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.815096    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.815179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.815115    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.815179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.815129    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.815179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.815145    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.815179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.815157    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.816547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.816519    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm" pod "ccd239f8-de79-43a0-af71-bf140440a576" (UID: "ccd239f8-de79-43a0-af71-bf140440a576"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.816639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.816579    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm" pod "669e0382-43ca-47fe-9926-4d9634ce133f" (UID: "669e0382-43ca-47fe-9926-4d9634ce133f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:04.816813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.816786    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs" pod "ccd239f8-de79-43a0-af71-bf140440a576" (UID: "ccd239f8-de79-43a0-af71-bf140440a576"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:04.816927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.816843    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx" pod "669e0382-43ca-47fe-9926-4d9634ce133f" (UID: "669e0382-43ca-47fe-9926-4d9634ce133f"). InnerVolumeSpecName "kube-api-access-fwkxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:04.816927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.816864    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc" pod "ccd239f8-de79-43a0-af71-bf140440a576" (UID: "ccd239f8-de79-43a0-af71-bf140440a576"). InnerVolumeSpecName "kube-api-access-n48nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:04.817077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.817062    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs" pod "669e0382-43ca-47fe-9926-4d9634ce133f" (UID: "669e0382-43ca-47fe-9926-4d9634ce133f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:04.916604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.916563    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkxx\" (UniqueName: \"kubernetes.io/projected/669e0382-43ca-47fe-9926-4d9634ce133f-kube-api-access-fwkxx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.916830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.916712    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd239f8-de79-43a0-af71-bf140440a576-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.916830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.916732    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd239f8-de79-43a0-af71-bf140440a576-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.916830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.916747    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669e0382-43ca-47fe-9926-4d9634ce133f-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.916830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.916761    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669e0382-43ca-47fe-9926-4d9634ce133f-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:04.916830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:04.916775    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n48nc\" (UniqueName: \"kubernetes.io/projected/ccd239f8-de79-43a0-af71-bf140440a576-kube-api-access-n48nc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:05.103796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.103698    5028 generic.go:356] "Generic (PLEG): container finished" podID="ccd239f8-de79-43a0-af71-bf140440a576" containerID="dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087" exitCode=0
Oct 30 22:40:05.103796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.103740    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" event={"ID":"ccd239f8-de79-43a0-af71-bf140440a576","Type":"ContainerDied","Data":"dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087"}
Oct 30 22:40:05.104055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.103799    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7" event={"ID":"ccd239f8-de79-43a0-af71-bf140440a576","Type":"ContainerDied","Data":"0cde920eabdff9481daf69a25c1fb4587a2f1f7ce935d08850e4d079a9fd43f0"}
Oct 30 22:40:05.104055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.103750    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"
Oct 30 22:40:05.104055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.103822    5028 scope.go:122] "RemoveContainer" containerID="dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087"
Oct 30 22:40:05.105444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.105411    5028 generic.go:356] "Generic (PLEG): container finished" podID="669e0382-43ca-47fe-9926-4d9634ce133f" containerID="139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754" exitCode=0
Oct 30 22:40:05.105573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.105457    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"
Oct 30 22:40:05.105573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.105514    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" event={"ID":"669e0382-43ca-47fe-9926-4d9634ce133f","Type":"ContainerDied","Data":"139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754"}
Oct 30 22:40:05.105573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.105546    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8" event={"ID":"669e0382-43ca-47fe-9926-4d9634ce133f","Type":"ContainerDied","Data":"ccd2294ba7911864efdd9cf6c7633f381da79484e038d58b55bcfa03eb0f6c86"}
Oct 30 22:40:05.116791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.116772    5028 scope.go:122] "RemoveContainer" containerID="dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087"
Oct 30 22:40:05.118003 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:05.117981    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087\": container with ID starting with dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087 not found: ID does not exist" containerID="dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087"
Oct 30 22:40:05.118113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.118009    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087"} err="failed to get container status \"dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087\": rpc error: code = NotFound desc = could not find container \"dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087\": container with ID starting with dc54a65a0fe44f7a8f54528a41f74474d371ed7e938e30f10607ae0de1bb6087 not found: ID does not exist"
Oct 30 22:40:05.118113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.118028    5028 scope.go:122] "RemoveContainer" containerID="139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754"
Oct 30 22:40:05.130937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.130906    5028 scope.go:122] "RemoveContainer" containerID="139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754"
Oct 30 22:40:05.131230 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:05.131205    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754\": container with ID starting with 139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754 not found: ID does not exist" containerID="139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754"
Oct 30 22:40:05.131301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.131238    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754"} err="failed to get container status \"139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754\": rpc error: code = NotFound desc = could not find container \"139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754\": container with ID starting with 139b8887680d4da723af1c2d3daaa6f65370c1508273f52f50cd82658d304754 not found: ID does not exist"
Oct 30 22:40:05.263605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.263562    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"]
Oct 30 22:40:05.267933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.267897    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-2-w0-d2d4c3-kserve-746d7b5f54-mh6j8"]
Oct 30 22:40:05.292596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.292564    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" path="/var/lib/kubelet/pods/669e0382-43ca-47fe-9926-4d9634ce133f/volumes"
Oct 30 22:40:05.372472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.372385    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"]
Oct 30 22:40:05.372642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.372591    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" containerID="cri-o://5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37" gracePeriod=30
Oct 30 22:40:05.377525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.377497    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"]
Oct 30 22:40:05.377764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.377731    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" containerID="cri-o://b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500" gracePeriod=30
Oct 30 22:40:05.382507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.382483    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"]
Oct 30 22:40:05.383239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.383200    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy" containerID="cri-o://dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b" gracePeriod=30
Oct 30 22:40:05.392195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.392160    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"]
Oct 30 22:40:05.421055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:05.421013    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/simulator-w0-d2d4c3-kserve-8464f7cf-zkjj7"]
Oct 30 22:40:06.202714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:06.202634    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.063412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.063356    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.278711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.273456    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.317802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.317122    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd239f8-de79-43a0-af71-bf140440a576" path="/var/lib/kubelet/pods/ccd239f8-de79-43a0-af71-bf140440a576/volumes"
Oct 30 22:40:07.436968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.436922    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.473873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.473538    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"]
Oct 30 22:40:07.475203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475173    5028 cpu_manager.go:414] "containerMap: removing container" podUID="57dfad38-3130-45a1-8ad7-5147d62da656" containerName="revoke-keys"
Oct 30 22:40:07.475409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475389    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dfad38-3130-45a1-8ad7-5147d62da656" containerName="revoke-keys"
Oct 30 22:40:07.475541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475529    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8ac38bc2-6733-4594-831a-e07764f2e3ce" containerName="revoke-keys"
Oct 30 22:40:07.475640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475618    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac38bc2-6733-4594-831a-e07764f2e3ce" containerName="revoke-keys"
Oct 30 22:40:07.475640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475640    5028 cpu_manager.go:414] "containerMap: removing container" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main"
Oct 30 22:40:07.475943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475650    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main"
Oct 30 22:40:07.475943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475784    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main"
Oct 30 22:40:07.475943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475795    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main"
Oct 30 22:40:07.475943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475815    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ceed8001-d99a-443e-b0de-1fb6c094b5e7" containerName="revoke-keys"
Oct 30 22:40:07.475943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.475822    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceed8001-d99a-443e-b0de-1fb6c094b5e7" containerName="revoke-keys"
Oct 30 22:40:07.476230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.476106    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd239f8-de79-43a0-af71-bf140440a576" containerName="main"
Oct 30 22:40:07.476230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.476126    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceed8001-d99a-443e-b0de-1fb6c094b5e7" containerName="revoke-keys"
Oct 30 22:40:07.476230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.476140    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dfad38-3130-45a1-8ad7-5147d62da656" containerName="revoke-keys"
Oct 30 22:40:07.476230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.476175    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac38bc2-6733-4594-831a-e07764f2e3ce" containerName="revoke-keys"
Oct 30 22:40:07.476230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.476186    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="669e0382-43ca-47fe-9926-4d9634ce133f" containerName="main"
Oct 30 22:40:07.483029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.483001    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.543192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.543158    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"]
Oct 30 22:40:07.571150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.571058    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:40:07.572068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.572029    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.572213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.572113    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.572213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.572180    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvpc\" (UniqueName: \"kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.573674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.573649    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:07.573841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.573741    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:07.574065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.574041    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"]
Oct 30 22:40:07.586125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.586093    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:07.590793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.589813    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.648341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.642783    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"]
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.650476    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"]
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.650854    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy" containerID="cri-o://cc275c72232d920e36051b3cfc878671cbfb9fbe8017ef892ac540ae8f853aac" gracePeriod=30
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.663782    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"]
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.664082    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy" containerID="cri-o://d30b3aec014d739a62f103c61fb6c49007d906613ef29e37256cf77e2ebc26d2" gracePeriod=30
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.668917    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"]
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.671798    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy" containerID="cri-o://e8391c72614d212e80b08e88442f9bcc5151d9d9194ba7df8929887329e28298" gracePeriod=30
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.674047    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.675162    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.675209    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.675238    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpl2k\" (UniqueName: \"kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.675296    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.675353    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.681709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.675396    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvpc\" (UniqueName: \"kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.700618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.700538    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.704710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.700876    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.717743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.711778    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"]
Oct 30 22:40:07.717743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.712022    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy" containerID="cri-o://0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d" gracePeriod=30
Oct 30 22:40:07.757537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.756648    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvpc\" (UniqueName: \"kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc\") pod \"maas-api-revoke-sub-5c099526-7n6kd\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.787102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.781633    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.787102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.781703    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpl2k\" (UniqueName: \"kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.787102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.781787    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.787102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.782847    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.799784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.799183    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.849659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.849565    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:07.849916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.849842    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:07.852206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.852178    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpl2k\" (UniqueName: \"kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k\") pod \"maas-api-revoke-sub-9afb2d3c-p84d7\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:07.937080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:07.936445    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:08.045710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:08.044267    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:08.482762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:08.474301    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:08.665120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:08.665086    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"]
Oct 30 22:40:08.672247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:08.671639    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"]
Oct 30 22:40:09.208101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.208067    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd" event={"ID":"87db2e32-aa79-409b-b1f9-2a9a50dc347e","Type":"ContainerStarted","Data":"8a29bc1ef724ec6588de3f8d916e674a1ddbc7b08376f8e4b2693d16edf83ba1"}
Oct 30 22:40:09.210275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.210233    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7" event={"ID":"058f959b-ad04-45b9-9162-f3392a85a7ad","Type":"ContainerStarted","Data":"2ad1bc5ae897e399446bba64144bd2135a06f30b898652653ce72b3b187c9bde"}
Oct 30 22:40:09.210408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.210301    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7" event={"ID":"058f959b-ad04-45b9-9162-f3392a85a7ad","Type":"ContainerStarted","Data":"410d53804da1b88c19b060f24e2175c4d4f3563e38f2d6cc362afb1e46c6144d"}
Oct 30 22:40:09.273542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.273507    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nwxll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:40:09.273698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.273568    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:40:09.347976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.345168    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7" podStartSLOduration=2.345147568 podStartE2EDuration="2.345147568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:09.337145155 +0000 UTC m=+2340.186658130" watchObservedRunningTime="2026-10-30 22:40:09.345147568 +0000 UTC m=+2340.194660542"
Oct 30 22:40:09.414256 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.414221    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:09.747576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.747539    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:09.768763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.768731    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-l597h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:40:09.768949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.768794    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:40:09.900472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:09.900445    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:10.248435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.248403    5028 generic.go:356] "Generic (PLEG): container finished" podID="058f959b-ad04-45b9-9162-f3392a85a7ad" containerID="2ad1bc5ae897e399446bba64144bd2135a06f30b898652653ce72b3b187c9bde" exitCode=0
Oct 30 22:40:10.248652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.248480    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7" event={"ID":"058f959b-ad04-45b9-9162-f3392a85a7ad","Type":"ContainerDied","Data":"2ad1bc5ae897e399446bba64144bd2135a06f30b898652653ce72b3b187c9bde"}
Oct 30 22:40:10.250439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.250414    5028 generic.go:356] "Generic (PLEG): container finished" podID="87db2e32-aa79-409b-b1f9-2a9a50dc347e" containerID="0654f497886b57ed57a0fb0f4ad1e2f73f179b585eb335e9b8c8db3f16c110b3" exitCode=0
Oct 30 22:40:10.250570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.250502    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd" event={"ID":"87db2e32-aa79-409b-b1f9-2a9a50dc347e","Type":"ContainerDied","Data":"0654f497886b57ed57a0fb0f4ad1e2f73f179b585eb335e9b8c8db3f16c110b3"}
Oct 30 22:40:10.763652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.763577    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:40:10.775579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.775545    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:40:10.854271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.854244    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:40:10.915618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915577    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.915861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915647    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.915861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915726    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.915861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915780    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.915861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915824    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.915861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915861    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915899    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915948    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.915984    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916056    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916095    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916094    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916131    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916142    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916192    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916226    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916272    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916315    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916356    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916384    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916411    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy\") pod \"9bb653cb-e45d-4edf-80f9-0514e2032093\" (UID: \"9bb653cb-e45d-4edf-80f9-0514e2032093\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916460    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916467    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.916858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.916753    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.928745    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.928833    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.928931    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.928971    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.929022    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.929068    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data\") pod \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\" (UID: \"c4cd3361-d3a9-4147-b1c4-5cddb8037c69\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.929179    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket\") pod \"8b99c592-742b-4b7d-a50e-70e079c7741e\" (UID: \"8b99c592-742b-4b7d-a50e-70e079c7741e\") "
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.930003    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb653cb-e45d-4edf-80f9-0514e2032093-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.930029    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.930044    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.934770    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.937925    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.943864    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.944223    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:10.944721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.944450    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.949354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.949321    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.949806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.949780    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.950033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.950012    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.951117    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.951431    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.951575    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.952174    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.953654    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.953931    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.954385    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.954462    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "kube-api-access-t8v8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.954986    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "kube-api-access-m492g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.955170    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "kube-api-access-w48v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:10.956748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.956635    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:10.957416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.956998    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token" pod "c4cd3361-d3a9-4147-b1c4-5cddb8037c69" (UID: "c4cd3361-d3a9-4147-b1c4-5cddb8037c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:10.957416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.957009    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy" pod "9bb653cb-e45d-4edf-80f9-0514e2032093" (UID: "9bb653cb-e45d-4edf-80f9-0514e2032093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:10.957416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.957095    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:10.957779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:10.957744    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy" pod "8b99c592-742b-4b7d-a50e-70e079c7741e" (UID: "8b99c592-742b-4b7d-a50e-70e079c7741e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049127    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049185    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049204    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8v8p\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-kube-api-access-t8v8p\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049219    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049253    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m492g\" (UniqueName: \"kubernetes.io/projected/8b99c592-742b-4b7d-a50e-70e079c7741e-kube-api-access-m492g\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049268    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049282    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049299    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049333    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049348    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049362    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049377    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049410    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049424    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049437    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049451    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049484    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049500    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049514    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48v7\" (UniqueName: \"kubernetes.io/projected/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-kube-api-access-w48v7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049530    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b99c592-742b-4b7d-a50e-70e079c7741e-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049561    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049575    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4cd3361-d3a9-4147-b1c4-5cddb8037c69-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049589    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb653cb-e45d-4edf-80f9-0514e2032093-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.049728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.049605    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b99c592-742b-4b7d-a50e-70e079c7741e-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.264936    5028 generic.go:356] "Generic (PLEG): container finished" podID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerID="dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b" exitCode=0
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.265003    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.265001    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" event={"ID":"9bb653cb-e45d-4edf-80f9-0514e2032093","Type":"ContainerDied","Data":"dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b"}
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.265145    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx" event={"ID":"9bb653cb-e45d-4edf-80f9-0514e2032093","Type":"ContainerDied","Data":"f5b9e05e9c621aeee02d69d3e743dafe8b282bcfd68adf8f580c0eeacea8c296"}
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.265201    5028 scope.go:122] "RemoveContainer" containerID="dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b"
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.266677    5028 generic.go:356] "Generic (PLEG): container finished" podID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerID="b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500" exitCode=0
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.266733    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.266741    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" event={"ID":"8b99c592-742b-4b7d-a50e-70e079c7741e","Type":"ContainerDied","Data":"b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500"}
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.266788    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h" event={"ID":"8b99c592-742b-4b7d-a50e-70e079c7741e","Type":"ContainerDied","Data":"15d1d3748f6e5aea6c78a166533783f40ea24496e4913283dc8d7131d3bbc84c"}
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.268583    5028 generic.go:356] "Generic (PLEG): container finished" podID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerID="5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37" exitCode=0
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.268744    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.268772    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" event={"ID":"c4cd3361-d3a9-4147-b1c4-5cddb8037c69","Type":"ContainerDied","Data":"5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37"}
Oct 30 22:40:11.269172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.268811    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll" event={"ID":"c4cd3361-d3a9-4147-b1c4-5cddb8037c69","Type":"ContainerDied","Data":"96664bf6aa906d3ab32cc69c67faa6b6906a3ecc962bd4da1d1c8a00c2fb3f3e"}
Oct 30 22:40:11.282627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.281960    5028 scope.go:122] "RemoveContainer" containerID="dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b"
Oct 30 22:40:11.282627 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:11.282314    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b\": container with ID starting with dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b not found: ID does not exist" containerID="dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b"
Oct 30 22:40:11.282627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.282354    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b"} err="failed to get container status \"dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b\": rpc error: code = NotFound desc = could not find container \"dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b\": container with ID starting with dae93aca1d59866f33607540f9f6620b40184d32e774c1b161b46da94d03216b not found: ID does not exist"
Oct 30 22:40:11.282627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.282467    5028 scope.go:122] "RemoveContainer" containerID="b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500"
Oct 30 22:40:11.303545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.303521    5028 scope.go:122] "RemoveContainer" containerID="b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500"
Oct 30 22:40:11.303915 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:11.303892    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500\": container with ID starting with b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500 not found: ID does not exist" containerID="b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500"
Oct 30 22:40:11.303961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.303924    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500"} err="failed to get container status \"b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500\": rpc error: code = NotFound desc = could not find container \"b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500\": container with ID starting with b6955b67bbfcf4e3e9e966fd7bab8bddc77dae36531f258bc4d05df9026a7500 not found: ID does not exist"
Oct 30 22:40:11.303961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.303947    5028 scope.go:122] "RemoveContainer" containerID="5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37"
Oct 30 22:40:11.342943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.342885    5028 scope.go:122] "RemoveContainer" containerID="5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37"
Oct 30 22:40:11.344715 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:11.343882    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37\": container with ID starting with 5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37 not found: ID does not exist" containerID="5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37"
Oct 30 22:40:11.344715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.343925    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37"} err="failed to get container status \"5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37\": rpc error: code = NotFound desc = could not find container \"5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37\": container with ID starting with 5e9e53d85f2274a3c12e749381ba7fbe82bb7d34f7c82aeeeeb39453e5d07f37 not found: ID does not exist"
Oct 30 22:40:11.462392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.462335    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"]
Oct 30 22:40:11.480428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.480285    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4xwjx"]
Oct 30 22:40:11.563806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.563737    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"]
Oct 30 22:40:11.564141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.564117    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:11.567239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.567220    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:11.568901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.568879    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-l597h"]
Oct 30 22:40:11.668806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.668762    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca\") pod \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") "
Oct 30 22:40:11.669014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.668822    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k\" (UniqueName: \"kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k\") pod \"058f959b-ad04-45b9-9162-f3392a85a7ad\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") "
Oct 30 22:40:11.669014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.668885    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca\") pod \"058f959b-ad04-45b9-9162-f3392a85a7ad\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") "
Oct 30 22:40:11.669014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.668946    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca\") pod \"058f959b-ad04-45b9-9162-f3392a85a7ad\" (UID: \"058f959b-ad04-45b9-9162-f3392a85a7ad\") "
Oct 30 22:40:11.669014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.668991    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca\") pod \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") "
Oct 30 22:40:11.669277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.669052    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc\" (UniqueName: \"kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc\") pod \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\" (UID: \"87db2e32-aa79-409b-b1f9-2a9a50dc347e\") "
Oct 30 22:40:11.669551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.669520    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca" pod "87db2e32-aa79-409b-b1f9-2a9a50dc347e" (UID: "87db2e32-aa79-409b-b1f9-2a9a50dc347e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:11.669658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.669549    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca" pod "058f959b-ad04-45b9-9162-f3392a85a7ad" (UID: "058f959b-ad04-45b9-9162-f3392a85a7ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:11.669795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.669772    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.669883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.669799    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.671450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.671407    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca" pod "87db2e32-aa79-409b-b1f9-2a9a50dc347e" (UID: "87db2e32-aa79-409b-b1f9-2a9a50dc347e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:11.671871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.671841    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k" pod "058f959b-ad04-45b9-9162-f3392a85a7ad" (UID: "058f959b-ad04-45b9-9162-f3392a85a7ad"). InnerVolumeSpecName "kube-api-access-gpl2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:11.672355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.671894    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca" pod "058f959b-ad04-45b9-9162-f3392a85a7ad" (UID: "058f959b-ad04-45b9-9162-f3392a85a7ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:11.672546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.672520    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc" pod "87db2e32-aa79-409b-b1f9-2a9a50dc347e" (UID: "87db2e32-aa79-409b-b1f9-2a9a50dc347e"). InnerVolumeSpecName "kube-api-access-zfvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:11.675594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.675566    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"]
Oct 30 22:40:11.680489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.680466    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nwxll"]
Oct 30 22:40:11.770620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.770577    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvpc\" (UniqueName: \"kubernetes.io/projected/87db2e32-aa79-409b-b1f9-2a9a50dc347e-kube-api-access-zfvpc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.770620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.770611    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87db2e32-aa79-409b-b1f9-2a9a50dc347e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.770620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.770624    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpl2k\" (UniqueName: \"kubernetes.io/projected/058f959b-ad04-45b9-9162-f3392a85a7ad-kube-api-access-gpl2k\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:11.771085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:11.770637    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058f959b-ad04-45b9-9162-f3392a85a7ad-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:12.280176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.280144    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"
Oct 30 22:40:12.280378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.280173    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7" event={"ID":"058f959b-ad04-45b9-9162-f3392a85a7ad","Type":"ContainerDied","Data":"410d53804da1b88c19b060f24e2175c4d4f3563e38f2d6cc362afb1e46c6144d"}
Oct 30 22:40:12.280378 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.280269    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410d53804da1b88c19b060f24e2175c4d4f3563e38f2d6cc362afb1e46c6144d"
Oct 30 22:40:12.281696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.281663    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"
Oct 30 22:40:12.281820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.281747    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd" event={"ID":"87db2e32-aa79-409b-b1f9-2a9a50dc347e","Type":"ContainerDied","Data":"8a29bc1ef724ec6588de3f8d916e674a1ddbc7b08376f8e4b2693d16edf83ba1"}
Oct 30 22:40:12.281820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.281787    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a29bc1ef724ec6588de3f8d916e674a1ddbc7b08376f8e4b2693d16edf83ba1"
Oct 30 22:40:12.706134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.706093    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:12.885971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:12.884433    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:13.075733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.075678    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:13.252469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.252346    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:13.302716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.302650    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:40:13.311046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.311011    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" path="/var/lib/kubelet/pods/8b99c592-742b-4b7d-a50e-70e079c7741e/volumes"
Oct 30 22:40:13.311790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.311768    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" path="/var/lib/kubelet/pods/9bb653cb-e45d-4edf-80f9-0514e2032093/volumes"
Oct 30 22:40:13.312446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.312426    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" path="/var/lib/kubelet/pods/c4cd3361-d3a9-4147-b1c4-5cddb8037c69/volumes"
Oct 30 22:40:13.338723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.338658    5028 generic.go:356] "Generic (PLEG): container finished" podID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerID="e8391c72614d212e80b08e88442f9bcc5151d9d9194ba7df8929887329e28298" exitCode=0
Oct 30 22:40:13.339297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.339271    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" event={"ID":"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8","Type":"ContainerDied","Data":"e8391c72614d212e80b08e88442f9bcc5151d9d9194ba7df8929887329e28298"}
Oct 30 22:40:13.341672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.341623    5028 generic.go:356] "Generic (PLEG): container finished" podID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerID="0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d" exitCode=0
Oct 30 22:40:13.342146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.342125    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" event={"ID":"b45f0344-ae7c-471a-9f98-cbde455efcda","Type":"ContainerDied","Data":"0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d"}
Oct 30 22:40:13.342337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.342322    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg" event={"ID":"b45f0344-ae7c-471a-9f98-cbde455efcda","Type":"ContainerDied","Data":"32ea90757261a4e9c3e8616f06d3cd41e3ad4241dcbed349d248000fcb464dc9"}
Oct 30 22:40:13.342528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.342493    5028 scope.go:122] "RemoveContainer" containerID="0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d"
Oct 30 22:40:13.342877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.342863    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"
Oct 30 22:40:13.347532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.347501    5028 generic.go:356] "Generic (PLEG): container finished" podID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerID="cc275c72232d920e36051b3cfc878671cbfb9fbe8017ef892ac540ae8f853aac" exitCode=0
Oct 30 22:40:13.347737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.347717    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" event={"ID":"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5","Type":"ContainerDied","Data":"cc275c72232d920e36051b3cfc878671cbfb9fbe8017ef892ac540ae8f853aac"}
Oct 30 22:40:13.350028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.349998    5028 generic.go:356] "Generic (PLEG): container finished" podID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerID="d30b3aec014d739a62f103c61fb6c49007d906613ef29e37256cf77e2ebc26d2" exitCode=0
Oct 30 22:40:13.350236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.350219    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" event={"ID":"7d252e95-ab99-45f6-8554-e0296a60de1a","Type":"ContainerDied","Data":"d30b3aec014d739a62f103c61fb6c49007d906613ef29e37256cf77e2ebc26d2"}
Oct 30 22:40:13.365267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.365244    5028 scope.go:122] "RemoveContainer" containerID="0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d"
Oct 30 22:40:13.365610 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:13.365584    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d\": container with ID starting with 0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d not found: ID does not exist" containerID="0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d"
Oct 30 22:40:13.365788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.365622    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d"} err="failed to get container status \"0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d\": rpc error: code = NotFound desc = could not find container \"0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d\": container with ID starting with 0448ad3ef199821184911c607718b5bad5d4c610ca34c318549115633281b52d not found: ID does not exist"
Oct 30 22:40:13.365788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.365644    5028 scope.go:122] "RemoveContainer" containerID="8c1b43780b2afafc3df4d78b7fd05bc8ca91bb2e49bf3f9b25dd8ec79cfbe3d0"
Oct 30 22:40:13.446204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446172    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446289    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446392    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446434    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446476    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446546    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446589    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446623    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.446992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.446652    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo\") pod \"b45f0344-ae7c-471a-9f98-cbde455efcda\" (UID: \"b45f0344-ae7c-471a-9f98-cbde455efcda\") "
Oct 30 22:40:13.448786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.447734    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.448786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.448055    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.448786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.448288    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.448786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.448503    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.449300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.449247    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:13.450230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.450197    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.450492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.450475    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:13.459276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.459196    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "kube-api-access-xl5jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.479901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.479865    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy" pod "b45f0344-ae7c-471a-9f98-cbde455efcda" (UID: "b45f0344-ae7c-471a-9f98-cbde455efcda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.484329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.484304    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:40:13.510216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.510070    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:13.527068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.527014    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:40:13.530535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.529606    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:40:13.547596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.547569    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.547814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.547794    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.547903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.547886    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.547966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.547929    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.548018 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.547961    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.548101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548087    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.548184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548171    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.548239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548200    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.548307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548294    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy\") pod \"7d252e95-ab99-45f6-8554-e0296a60de1a\" (UID: \"7d252e95-ab99-45f6-8554-e0296a60de1a\") "
Oct 30 22:40:13.548473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548444    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.548743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548716    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.548956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.548926    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.549213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549193    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549341    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549366    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549380    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549395    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5jn\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-kube-api-access-xl5jn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549409    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549425    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549439    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549453    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45f0344-ae7c-471a-9f98-cbde455efcda-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549466    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549480    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549493    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45f0344-ae7c-471a-9f98-cbde455efcda-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549245    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:13.549538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.549471    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.550933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.550907    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "kube-api-access-8h5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.551472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.551449    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:13.551666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.551637    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.551801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.551784    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token" pod "7d252e95-ab99-45f6-8554-e0296a60de1a" (UID: "7d252e95-ab99-45f6-8554-e0296a60de1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.690710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.657145    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.691039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.691010    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.691194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.691178    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.691316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.691301    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.713243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.691462    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.692704    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.693026    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.693646    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715125    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715205    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715249    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715310    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715385    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715453    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715525    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715557    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715649    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo\") pod \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\" (UID: \"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715734    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715824    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715881    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.716305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.715952    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy\") pod \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\" (UID: \"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5\") "
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716661    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716704    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5ps\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-kube-api-access-8h5ps\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716721    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716736    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716764    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716804    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716818    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d252e95-ab99-45f6-8554-e0296a60de1a-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716832    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d252e95-ab99-45f6-8554-e0296a60de1a-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.717324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.716845    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d252e95-ab99-45f6-8554-e0296a60de1a-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.718489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.718304    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.719086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.718799    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:13.719355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.719331    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "kube-api-access-djtdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.720908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.719953    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.720908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.720076    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.720908 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.720150    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.721196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.721171    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:13.721255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.721238    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.722165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.722137    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "kube-api-access-zhcgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.730033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.723487    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.730033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.725667    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:13.730033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.726417    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.730033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.729277    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:13.730033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.729759    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo" pod "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" (UID: "cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:13.730904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.730779    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token" pod "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" (UID: "4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817837    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817865    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817876    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817884    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817893    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817902    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817911    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817919    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.817918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817927    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.818535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817935    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.818535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817944    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtdt\" (UniqueName: \"kubernetes.io/projected/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-kube-api-access-djtdt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.818535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817954    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcgs\" (UniqueName: \"kubernetes.io/projected/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-kube-api-access-zhcgs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.818535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817962    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.818535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817970    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.818535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:13.817979    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:14.008786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.008705    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"]
Oct 30 22:40:14.017132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.017091    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ssfrg"]
Oct 30 22:40:14.364679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.364638    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr" event={"ID":"cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5","Type":"ContainerDied","Data":"8e22895d7d1e78c82f41ace1c81873dcc5cf245b396a17847dc94efae302f266"}
Oct 30 22:40:14.364679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.364660    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"
Oct 30 22:40:14.364974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.364750    5028 scope.go:122] "RemoveContainer" containerID="cc275c72232d920e36051b3cfc878671cbfb9fbe8017ef892ac540ae8f853aac"
Oct 30 22:40:14.368932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.366940    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"
Oct 30 22:40:14.368932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.366936    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc" event={"ID":"7d252e95-ab99-45f6-8554-e0296a60de1a","Type":"ContainerDied","Data":"19709f5fc52c8311386096705fca9d927511f833fc1dbc77d3fae45710074ef6"}
Oct 30 22:40:14.381881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.370124    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h" event={"ID":"4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8","Type":"ContainerDied","Data":"3b82e7ad9c147319252260927d69619d72a51a280cb32ed92baf240ca6a5c5d4"}
Oct 30 22:40:14.381881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.370137    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"
Oct 30 22:40:14.381881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.381400    5028 scope.go:122] "RemoveContainer" containerID="d30b3aec014d739a62f103c61fb6c49007d906613ef29e37256cf77e2ebc26d2"
Oct 30 22:40:14.395776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.395758    5028 scope.go:122] "RemoveContainer" containerID="e8391c72614d212e80b08e88442f9bcc5151d9d9194ba7df8929887329e28298"
Oct 30 22:40:14.443618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.443590    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:14.526139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.526041    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"]
Oct 30 22:40:14.531143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.531113    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-zgcwc"]
Oct 30 22:40:14.638357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.638317    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"]
Oct 30 22:40:14.643458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.643432    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rjp6h"]
Oct 30 22:40:14.729102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.729064    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:14.750516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.750483    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"]
Oct 30 22:40:14.757260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.757230    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-s6krr"]
Oct 30 22:40:14.877346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:14.877261    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:15.000872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.000835    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"]
Oct 30 22:40:15.004769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.004742    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f80a0e-9vmz4"]
Oct 30 22:40:15.056922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.056878    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:40:15.057351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.056953    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:40:15.296312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.296276    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" path="/var/lib/kubelet/pods/4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8/volumes"
Oct 30 22:40:15.296925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.296901    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" path="/var/lib/kubelet/pods/7d252e95-ab99-45f6-8554-e0296a60de1a/volumes"
Oct 30 22:40:15.297437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.297416    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" path="/var/lib/kubelet/pods/b45f0344-ae7c-471a-9f98-cbde455efcda/volumes"
Oct 30 22:40:15.298241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.298218    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98d6477-d401-4cb3-ac4e-be00821a7fbe" path="/var/lib/kubelet/pods/c98d6477-d401-4cb3-ac4e-be00821a7fbe/volumes"
Oct 30 22:40:15.298766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.298748    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" path="/var/lib/kubelet/pods/cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5/volumes"
Oct 30 22:40:15.447431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:15.447389    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:16.935044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:16.934991    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:17.112967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.112929    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:17.411309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.411276    5028 generic.go:356] "Generic (PLEG): container finished" podID="8791da6c-589d-4f64-b334-bc44be58efb7" containerID="06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816" exitCode=0
Oct 30 22:40:17.411529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.411345    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" event={"ID":"8791da6c-589d-4f64-b334-bc44be58efb7","Type":"ContainerDied","Data":"06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"}
Oct 30 22:40:17.411798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.411783    5028 scope.go:122] "RemoveContainer" containerID="06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"
Oct 30 22:40:17.413495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.413468    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:17.413566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.413538    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:17.413603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.413570    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:17.517607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.517575    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"]
Oct 30 22:40:17.518486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518462    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy"
Oct 30 22:40:17.518486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518486    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518511    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518519    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518545    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518554    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518586    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518593    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518614    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518622    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518634    5028 cpu_manager.go:414] "containerMap: removing container" podUID="87db2e32-aa79-409b-b1f9-2a9a50dc347e" containerName="revoke-keys"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518642    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87db2e32-aa79-409b-b1f9-2a9a50dc347e" containerName="revoke-keys"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518663    5028 cpu_manager.go:414] "containerMap: removing container" podUID="058f959b-ad04-45b9-9162-f3392a85a7ad" containerName="revoke-keys"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518671    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058f959b-ad04-45b9-9162-f3392a85a7ad" containerName="revoke-keys"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518735    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy"
Oct 30 22:40:17.518757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518747    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518766    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518774    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518785    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518793    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518951    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518968    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0ed72a-3967-4cc3-8173-4a0c1fccd7f8" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518982    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cd3361-d3a9-4147-b1c4-5cddb8037c69" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.518997    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45f0344-ae7c-471a-9f98-cbde455efcda" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.519008    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9f01fc-ffcb-4164-a4c9-a4cd0c9acef5" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.519017    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="87db2e32-aa79-409b-b1f9-2a9a50dc347e" containerName="revoke-keys"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.519032    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="058f959b-ad04-45b9-9162-f3392a85a7ad" containerName="revoke-keys"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.519041    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b99c592-742b-4b7d-a50e-70e079c7741e" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.519052    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb653cb-e45d-4edf-80f9-0514e2032093" containerName="istio-proxy"
Oct 30 22:40:17.519980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.519063    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d252e95-ab99-45f6-8554-e0296a60de1a" containerName="istio-proxy"
Oct 30 22:40:17.523588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.523519    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.596862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.584739    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.596862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.584819    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swh7\" (UniqueName: \"kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7\") pod \"payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.598218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.598189    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9fce61\""
Oct 30 22:40:17.598982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.598932    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9fce61-dockercfg-qlbdc\""
Oct 30 22:40:17.611526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.602470    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"]
Oct 30 22:40:17.662483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.662446    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"]
Oct 30 22:40:17.668262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.668233    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.699978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.699895    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.700335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.700303    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swh7\" (UniqueName: \"kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7\") pod \"payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.701072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.701047    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.721855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.721822    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"]
Oct 30 22:40:17.741517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.741470    5028 scope.go:122] "RemoveContainer" containerID="81c443ece20f63ca06ebe0e56d4f541f93c9629f3fd213b1bb8259b1d69eb287"
Oct 30 22:40:17.755070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.755036    5028 scope.go:122] "RemoveContainer" containerID="55762b9b2b0acca1d13951820d5522b2041bb590a24035823c1ff5b3fada9faf"
Oct 30 22:40:17.761294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.761265    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swh7\" (UniqueName: \"kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7\") pod \"payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.801592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.801560    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvxm\" (UniqueName: \"kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm\") pod \"payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.801787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.801682    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.845863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.845836    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:17.902673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.902642    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvxm\" (UniqueName: \"kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm\") pod \"payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.904375 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.902775    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.904375 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.903436    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.962842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.962811    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvxm\" (UniqueName: \"kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm\") pod \"payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:17.988366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:17.988067    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:18.256046 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:40:18.255999    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd292b8_d597_4f36_a8a0_fff7a36e94c9.slice/crio-8e0f5b8ba7bf6ea1b98732acdf25cfee310586e0fe46fa2b0ec60124e42209f4 WatchSource:0}: Error finding container 8e0f5b8ba7bf6ea1b98732acdf25cfee310586e0fe46fa2b0ec60124e42209f4: Status 404 returned error can't find the container with id 8e0f5b8ba7bf6ea1b98732acdf25cfee310586e0fe46fa2b0ec60124e42209f4
Oct 30 22:40:18.256403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.256381    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"]
Oct 30 22:40:18.379097 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:40:18.379063    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27c8960_61d7_4534_bede_92c649f1b644.slice/crio-868c1655ed326f71bf3a3679bc7e3d75bb1a6c5e1f12359def153dd4c9a9c822 WatchSource:0}: Error finding container 868c1655ed326f71bf3a3679bc7e3d75bb1a6c5e1f12359def153dd4c9a9c822: Status 404 returned error can't find the container with id 868c1655ed326f71bf3a3679bc7e3d75bb1a6c5e1f12359def153dd4c9a9c822
Oct 30 22:40:18.379737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.379708    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"]
Oct 30 22:40:18.444593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.444550    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8" event={"ID":"f27c8960-61d7-4534-bede-92c649f1b644","Type":"ContainerStarted","Data":"868c1655ed326f71bf3a3679bc7e3d75bb1a6c5e1f12359def153dd4c9a9c822"}
Oct 30 22:40:18.447912 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.447879    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" event={"ID":"8791da6c-589d-4f64-b334-bc44be58efb7","Type":"ContainerStarted","Data":"53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae"}
Oct 30 22:40:18.449900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.449867    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6" event={"ID":"5dd292b8-d597-4f36-a8a0-fff7a36e94c9","Type":"ContainerStarted","Data":"b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9"}
Oct 30 22:40:18.450016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.449921    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6" event={"ID":"5dd292b8-d597-4f36-a8a0-fff7a36e94c9","Type":"ContainerStarted","Data":"8e0f5b8ba7bf6ea1b98732acdf25cfee310586e0fe46fa2b0ec60124e42209f4"}
Oct 30 22:40:18.450016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.449956    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:18.676969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:18.676868    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6" podStartSLOduration=1.676854558 podStartE2EDuration="1.676854558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:18.674734336 +0000 UTC m=+2349.524247309" watchObservedRunningTime="2026-10-30 22:40:18.676854558 +0000 UTC m=+2349.526367531"
Oct 30 22:40:19.126720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.126349    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:40:19.135709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.134835    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:40:19.135709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.134896    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:40:19.413062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.412973    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:19.461279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.461241    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8" event={"ID":"f27c8960-61d7-4534-bede-92c649f1b644","Type":"ContainerStarted","Data":"5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea"}
Oct 30 22:40:19.461587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.461566    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:19.574851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.574797    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8" podStartSLOduration=2.574781852 podStartE2EDuration="2.574781852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:19.571984931 +0000 UTC m=+2350.421497905" watchObservedRunningTime="2026-10-30 22:40:19.574781852 +0000 UTC m=+2350.424294858"
Oct 30 22:40:19.601098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:19.601069    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:20.125712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:20.123191    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:40:20.125712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:20.123259    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:40:21.126178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.126138    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:40:21.127593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.127558    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:40:21.482210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.482172    5028 generic.go:356] "Generic (PLEG): container finished" podID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerID="3a58f9b69f35ff7b26879846ffed044a67d2acccb1ed0a0a99f3724533d437cd" exitCode=0
Oct 30 22:40:21.482364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.482288    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" event={"ID":"f5ec1f18-325e-4a7e-b198-8be5c306938d","Type":"ContainerDied","Data":"3a58f9b69f35ff7b26879846ffed044a67d2acccb1ed0a0a99f3724533d437cd"}
Oct 30 22:40:21.691211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.691140    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:40:21.885613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.885584    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7\" (UniqueName: \"kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7\") pod \"f5ec1f18-325e-4a7e-b198-8be5c306938d\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") "
Oct 30 22:40:21.885848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.885833    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume\") pod \"f5ec1f18-325e-4a7e-b198-8be5c306938d\" (UID: \"f5ec1f18-325e-4a7e-b198-8be5c306938d\") "
Oct 30 22:40:21.886203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.886175    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume" pod "f5ec1f18-325e-4a7e-b198-8be5c306938d" (UID: "f5ec1f18-325e-4a7e-b198-8be5c306938d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:21.887723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.887672    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7" pod "f5ec1f18-325e-4a7e-b198-8be5c306938d" (UID: "f5ec1f18-325e-4a7e-b198-8be5c306938d"). InnerVolumeSpecName "kube-api-access-gxqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:21.943340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.943235    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:21.987914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.987880    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqj7\" (UniqueName: \"kubernetes.io/projected/f5ec1f18-325e-4a7e-b198-8be5c306938d-kube-api-access-gxqj7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:21.987914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:21.987911    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec1f18-325e-4a7e-b198-8be5c306938d-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:22.140712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.140318    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:40:22.140712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.140399    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:40:22.242787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.242667    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:22.399102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.398995    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:22.504636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.504533    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7" event={"ID":"f5ec1f18-325e-4a7e-b198-8be5c306938d","Type":"ContainerDied","Data":"a0ade28a5e8bf95e698bbed67731aeec7e31a6797f400c54556f77e1db81780d"}
Oct 30 22:40:22.504636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.504606    5028 scope.go:122] "RemoveContainer" containerID="3a58f9b69f35ff7b26879846ffed044a67d2acccb1ed0a0a99f3724533d437cd"
Oct 30 22:40:22.504895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.504821    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"
Oct 30 22:40:22.664606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.664569    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"]
Oct 30 22:40:22.711237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:22.711207    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d2d4c3-847ddbf687-jwnm7"]
Oct 30 22:40:23.123167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:23.123129    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:40:23.123456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:23.123430    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:40:23.292731 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:23.292675    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" path="/var/lib/kubelet/pods/f5ec1f18-325e-4a7e-b198-8be5c306938d/volumes"
Oct 30 22:40:24.126736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:24.126698    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:40:24.127246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:24.127220    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:40:24.127910 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:24.127893    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:40:24.211343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:24.211309    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:26.565765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:26.565726    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:26.714780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:26.714714    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:27.318708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:27.318634    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:28.437199 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:28.437157    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:40:28.831880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:28.831805    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:29.466200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:29.466168    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:40:30.476295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:30.476266    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:40:33.321135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:33.321096    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:33.580025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:33.579938    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:38.051024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:38.050986    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:39.832791 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:39.832744    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:42.343630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:42.343592    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:43.819028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:43.818985    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:44.051142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:44.051109    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:44.204747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:44.204639    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:44.336653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:44.336615    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"]
Oct 30 22:40:44.382177 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:44.382143    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-af30a9-58mqz"]
Oct 30 22:40:44.829722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:44.829674    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:45.292777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:45.292728    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2efc9b-59a4-409f-9e2b-7b9be4c72e70" path="/var/lib/kubelet/pods/9f2efc9b-59a4-409f-9e2b-7b9be4c72e70/volumes"
Oct 30 22:40:48.941954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:48.941919    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:50.894956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:50.894920    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:51.046010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:51.045963    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:51.265807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:51.265776    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:52.935041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:52.935002    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:53.858779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:53.858742    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:57.255312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.255271    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"]
Oct 30 22:40:57.255956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.255919    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerName="payload-processing"
Oct 30 22:40:57.255956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.255947    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerName="payload-processing"
Oct 30 22:40:57.256185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.256169    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ec1f18-325e-4a7e-b198-8be5c306938d" containerName="payload-processing"
Oct 30 22:40:57.261235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.261211    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.302117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.302081    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"]
Oct 30 22:40:57.306738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.306680    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.323357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.323320    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"]
Oct 30 22:40:57.358818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.358778    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"]
Oct 30 22:40:57.411510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411469    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.411759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411528    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.411759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411612    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.411759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411641    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.411759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411665    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.411996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411752    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.411996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411799    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.411996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411830    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.411996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411904    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z84j\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.411996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411954    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.411996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.411985    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.412248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412009    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.412248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412046    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.412248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412094    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.412248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412118    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.412248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412167    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.412248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412222    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blfv\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.412442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.412289    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513361    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z84j\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513406    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513424    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.513454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513441    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513456    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513483    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513516    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513561    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513619    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blfv\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513657    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513808    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513862    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.513921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513898    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513934    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513959    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513955    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513989    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513998    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.514007    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.513998    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.514598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.514296    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.515003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.514985    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.515233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.515175    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.515233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.515204    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.515438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.515276    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.515438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.515318    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.515438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.515207    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.515750 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.515726    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.516643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.516615    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.517136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.517109    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.517257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.517138    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.517487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.517470    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.572646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.572612    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.572846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.572658    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blfv\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.572898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.572883    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z84j\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.573810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.573791    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:57.576801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.576782    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:57.626881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:57.626574    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:58.007711 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:40:58.007667    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5232f8_2c29_407f_b212_917e30f59647.slice/crio-0db06371f3f437d3e46b3c3cf079572a991cd0f9d26aa8f810ce011c97283114 WatchSource:0}: Error finding container 0db06371f3f437d3e46b3c3cf079572a991cd0f9d26aa8f810ce011c97283114: Status 404 returned error can't find the container with id 0db06371f3f437d3e46b3c3cf079572a991cd0f9d26aa8f810ce011c97283114
Oct 30 22:40:58.008422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.008385    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"]
Oct 30 22:40:58.010495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.010459    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:58.010610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.010539    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:58.010610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.010585    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:58.058631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.058601    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"]
Oct 30 22:40:58.059232 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:40:58.059211    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde55b3ff_0808_4f8d_95a2_8e8f046af5d7.slice/crio-4facf05295cee29bfceee6a7c37aee9fdd1d7295b105a371e4dda462b45b18e0 WatchSource:0}: Error finding container 4facf05295cee29bfceee6a7c37aee9fdd1d7295b105a371e4dda462b45b18e0: Status 404 returned error can't find the container with id 4facf05295cee29bfceee6a7c37aee9fdd1d7295b105a371e4dda462b45b18e0
Oct 30 22:40:58.062317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.062284    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:58.062411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.062356    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:58.062411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.062385    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:40:58.352961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.352859    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:58.446151 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:40:58.445436    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:40:58.874802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.874755    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" event={"ID":"ad5232f8-2c29-407f-b212-917e30f59647","Type":"ContainerStarted","Data":"e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975"}
Oct 30 22:40:58.875029 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.874815    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" event={"ID":"ad5232f8-2c29-407f-b212-917e30f59647","Type":"ContainerStarted","Data":"0db06371f3f437d3e46b3c3cf079572a991cd0f9d26aa8f810ce011c97283114"}
Oct 30 22:40:58.876739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.876712    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" event={"ID":"de55b3ff-0808-4f8d-95a2-8e8f046af5d7","Type":"ContainerStarted","Data":"a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427"}
Oct 30 22:40:58.876884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:58.876756    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" event={"ID":"de55b3ff-0808-4f8d-95a2-8e8f046af5d7","Type":"ContainerStarted","Data":"4facf05295cee29bfceee6a7c37aee9fdd1d7295b105a371e4dda462b45b18e0"}
Oct 30 22:40:59.042714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.039339    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podStartSLOduration=2.039317053 podStartE2EDuration="2.039317053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:59.007990039 +0000 UTC m=+2389.857503013" watchObservedRunningTime="2026-10-30 22:40:59.039317053 +0000 UTC m=+2389.888830027"
Oct 30 22:40:59.577007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.576964    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:40:59.578625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.578596    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:40:59.578775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.578664    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 22:40:59.647542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.647507    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:40:59.654724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.653859    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 22:40:59.654724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:40:59.653939    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 22:41:00.578164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:00.578133    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:41:00.578563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:00.578198    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 22:41:00.634318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:00.634194    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 22:41:00.634318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:00.634256    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 22:41:01.578617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:01.578588    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:41:01.579127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:01.578810    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 22:41:01.650713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:01.643851    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 22:41:01.650713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:01.643915    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 22:41:02.577443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:02.577412    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:41:02.577668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:02.577467    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 22:41:02.646712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:02.639756    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 22:41:02.646712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:02.639813    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 22:41:02.787875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:02.786176    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:03.072699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:03.072645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:03.577749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:03.577714    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:41:03.577949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:03.577785    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 22:41:03.633714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:03.631032    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 22:41:03.633714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:03.631092    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 22:41:04.577343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:04.577300    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:41:04.577777 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:04.577372    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Oct 30 22:41:04.627429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:04.627389    5028 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Oct 30 22:41:04.627609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:04.627465    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Oct 30 22:41:05.583840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.583806    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:41:05.584342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.584287    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:41:05.585089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.585071    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:41:05.633331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.633297    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:41:05.633604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.633585    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:41:05.634313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.634295    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:41:05.748889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.748832    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podStartSLOduration=8.748818854 podStartE2EDuration="8.748818854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:59.148107034 +0000 UTC m=+2389.997620008" watchObservedRunningTime="2026-10-30 22:41:05.748818854 +0000 UTC m=+2396.598331826"
Oct 30 22:41:05.891637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.891555    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"]
Oct 30 22:41:05.896031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.895996    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:05.911218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.911184    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"]
Oct 30 22:41:05.917251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.917210    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:05.985458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:05.984728    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"]
Oct 30 22:41:06.036231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036191    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.036455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036255    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.036455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036287    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.036455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036370    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.036625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036496    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.036625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036546    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.036625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036618    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.036832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036674    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.036832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036738    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.036832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036787    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.036986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.036853    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh894\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.037038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037024    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.037093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037060    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.037466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037408    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.037466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037457    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.037732 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037699    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.037831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037753    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.037904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.037869    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thkb6\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.047854 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.047815    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"]
Oct 30 22:41:06.139398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139364    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.139566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139420    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh894\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.139566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139454    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.139566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139472    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.139789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139563    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.139789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139603    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.139789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139634    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.139789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139660    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.139789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139735    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thkb6\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.139789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139783    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139823    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139849    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139877    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139888    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139922    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139953    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140004    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140068    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140097    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140133    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.139950    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140294    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140320    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.140786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140476    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.140786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140571    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.141067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.140878    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.141067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.141025    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.141196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.141180    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.142432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.142371    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.142529 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.142516    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.142958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.142940    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.143025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.142986    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.198624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.198590    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.198790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.198661    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.198902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.198880    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh894\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zldcw\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.198942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.198880    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thkb6\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-9qn88\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.228388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.228352    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:06.236873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.236844    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:06.659438 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:41:06.659393    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8720c1_5f98_4e9a_acc8_a65378d1b743.slice/crio-74371c85b4ebd1003fee8ef28bc84db89decfed836ac7d55fca804ad8543aeb2 WatchSource:0}: Error finding container 74371c85b4ebd1003fee8ef28bc84db89decfed836ac7d55fca804ad8543aeb2: Status 404 returned error can't find the container with id 74371c85b4ebd1003fee8ef28bc84db89decfed836ac7d55fca804ad8543aeb2
Oct 30 22:41:06.660272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.660248    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"]
Oct 30 22:41:06.662357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.662321    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:41:06.662453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.662391    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:41:06.662453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.662419    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:41:06.715095 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:41:06.715067    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d3a0dc_cc45_4619_bb5b_da11755ba935.slice/crio-af78e0231d20a9560c7dad6a32cc34e28310a8951afa7ff66380ab204e0c3c37 WatchSource:0}: Error finding container af78e0231d20a9560c7dad6a32cc34e28310a8951afa7ff66380ab204e0c3c37: Status 404 returned error can't find the container with id af78e0231d20a9560c7dad6a32cc34e28310a8951afa7ff66380ab204e0c3c37
Oct 30 22:41:06.717971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.717944    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"]
Oct 30 22:41:06.718675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.718639    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:41:06.718811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.718734    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:41:06.718811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.718783    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:41:06.971208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.970494    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" event={"ID":"2d8720c1-5f98-4e9a-acc8-a65378d1b743","Type":"ContainerStarted","Data":"59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39"}
Oct 30 22:41:06.971208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.970561    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" event={"ID":"2d8720c1-5f98-4e9a-acc8-a65378d1b743","Type":"ContainerStarted","Data":"74371c85b4ebd1003fee8ef28bc84db89decfed836ac7d55fca804ad8543aeb2"}
Oct 30 22:41:06.972559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.972523    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" event={"ID":"78d3a0dc-cc45-4619-bb5b-da11755ba935","Type":"ContainerStarted","Data":"56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524"}
Oct 30 22:41:06.972757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:06.972593    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" event={"ID":"78d3a0dc-cc45-4619-bb5b-da11755ba935","Type":"ContainerStarted","Data":"af78e0231d20a9560c7dad6a32cc34e28310a8951afa7ff66380ab204e0c3c37"}
Oct 30 22:41:07.088353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.088298    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podStartSLOduration=2.088284136 podStartE2EDuration="2.088284136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:07.086257752 +0000 UTC m=+2397.935770751" watchObservedRunningTime="2026-10-30 22:41:07.088284136 +0000 UTC m=+2397.937797106"
Oct 30 22:41:07.201789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.201673    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podStartSLOduration=2.201652073 podStartE2EDuration="2.201652073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:07.201009946 +0000 UTC m=+2398.050522920" watchObservedRunningTime="2026-10-30 22:41:07.201652073 +0000 UTC m=+2398.051165046"
Oct 30 22:41:07.229626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.229523    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:07.230820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.230795    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:07.230932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.230844    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:07.243446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.243411    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:07.243810 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.243788    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:07.243963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.243941    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:07.571794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:07.571646    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:08.245168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:08.245123    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:08.245839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:08.245805    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:08.246069 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:08.246039    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:08.246309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:08.246285    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:08.588875 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:08.588789    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:08.753726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:08.750402    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:09.232713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:09.231208    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:09.232713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:09.231270    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:09.248431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:09.241711    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:09.248431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:09.241788    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:10.233712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:10.229168    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:10.233712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:10.229232    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:10.242713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:10.240145    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:10.242713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:10.240210    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:10.402786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:10.402759    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:41:10.440722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:10.438729    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:41:11.238718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:11.230084    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:11.238718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:11.230152    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:11.242711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:11.239111    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:11.242711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:11.239187    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:12.177726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.177027    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:12.240769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.229370    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:12.240769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.229444    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:12.240769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.239095    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:12.240769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.239148    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:12.380290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.380255    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:12.576730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:12.574286    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.229406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:13.229363    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:13.229930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:13.229439    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:13.237911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:13.237869    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:13.238064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:13.237929    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:13.583136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:13.583020    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:14.229262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:14.229217    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zldcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:41:14.229521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:14.229290    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 30 22:41:14.237934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:14.237880    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-9qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:41:14.238138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:14.237960    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 22:41:15.699196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:15.699162    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:15.701391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:15.701315    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:15.703781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:15.703679    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:41:15.806866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:15.806828    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:15.807632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:15.807601    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:15.808193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:15.808176    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:41:16.372896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:16.372864    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:17.787469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:17.787428    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:17.907786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:17.907754    5028 scope.go:122] "RemoveContainer" containerID="d317f26fcc8a2abd58dd4ad4f24d4ceb211e7c3ec9838ab01f6c937937d1fc9b"
Oct 30 22:41:17.972758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:17.972723    5028 scope.go:122] "RemoveContainer" containerID="c2af00d1a3252ca33efd9daced6381a2bc6f961cebc87faae9bb79ab8a0d1034"
Oct 30 22:41:18.003509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.003484    5028 scope.go:122] "RemoveContainer" containerID="d9016e1bfa9a5e84a9d4dd38fadb6d734199506545875427f2b796634db0c50a"
Oct 30 22:41:18.004248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.004038    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:18.015879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.015863    5028 scope.go:122] "RemoveContainer" containerID="839201a53e624e6a5a3e118c9faaefc1f599b42c0f2d3632b63ae564c95a7789"
Oct 30 22:41:18.174155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.173774    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:18.270708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.270310    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"]
Oct 30 22:41:18.278071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.277955    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d2d4c3-7kcf5"]
Oct 30 22:41:18.711632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:18.711591    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:19.294043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:19.294006    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceed8001-d99a-443e-b0de-1fb6c094b5e7" path="/var/lib/kubelet/pods/ceed8001-d99a-443e-b0de-1fb6c094b5e7/volumes"
Oct 30 22:41:22.802711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:22.797877    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:25.000957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.000921    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:25.159986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.159952    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:25.375360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.375269    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"]
Oct 30 22:41:25.375578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.375548    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy" containerID="cri-o://f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc" gracePeriod=30
Oct 30 22:41:25.385471 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.385432    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"]
Oct 30 22:41:25.385714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.385674    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy" containerID="cri-o://e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975" gracePeriod=30
Oct 30 22:41:25.390670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.390639    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:25.411522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.411491    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"]
Oct 30 22:41:25.411743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:25.411715    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy" containerID="cri-o://a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427" gracePeriod=30
Oct 30 22:41:27.249465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:27.249424    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:28.437242 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:41:28.437203    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:41:30.714308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.714276    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:41:30.739252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.739224    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:41:30.790800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790767    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.790982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790808    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.790982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790865    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.790982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790882    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.790982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790922    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.790982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790962    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.790993    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791052    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791078    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.791262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791100    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791149    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791284    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791349    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791388    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791511    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791531    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791567    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.791643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791598    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.792070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791651    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket\") pod \"ad5232f8-2c29-407f-b212-917e30f59647\" (UID: \"ad5232f8-2c29-407f-b212-917e30f59647\") "
Oct 30 22:41:30.792070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791724    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.792070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791766    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token\") pod \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\" (UID: \"de55b3ff-0808-4f8d-95a2-8e8f046af5d7\") "
Oct 30 22:41:30.792070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.791946    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:30.792287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792230    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.792287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792247    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.792287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792256    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792479    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792552    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792574    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792591    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792604    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792619    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792632    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792648    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5232f8-2c29-407f-b212-917e30f59647-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792649    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.792678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.792651    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.793906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.793843    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "kube-api-access-9blfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:30.794065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.794025    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "kube-api-access-9z84j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:30.794065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.794026    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:30.794227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.794119    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.794623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.794596    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:30.794771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.794752    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:30.794838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.794790    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token" pod "ad5232f8-2c29-407f-b212-917e30f59647" (UID: "ad5232f8-2c29-407f-b212-917e30f59647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:30.795268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.795244    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy" pod "de55b3ff-0808-4f8d-95a2-8e8f046af5d7" (UID: "de55b3ff-0808-4f8d-95a2-8e8f046af5d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.894850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894812    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.894850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894851    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894867    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894882    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894897    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894911    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894925    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5232f8-2c29-407f-b212-917e30f59647-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894941    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z84j\" (UniqueName: \"kubernetes.io/projected/ad5232f8-2c29-407f-b212-917e30f59647-kube-api-access-9z84j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894956    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blfv\" (UniqueName: \"kubernetes.io/projected/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-kube-api-access-9blfv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894970    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5232f8-2c29-407f-b212-917e30f59647-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.895048 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:30.894985    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de55b3ff-0808-4f8d-95a2-8e8f046af5d7-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.045146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.045115    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:41:31.103941 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.103910    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.103961    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.103998    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104176    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104240    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104291    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:31.104414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104305    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104364    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104439    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:31.104643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104521    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104585    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket\") pod \"b242cced-73d5-48cf-9819-84803277f9da\" (UID: \"b242cced-73d5-48cf-9819-84803277f9da\") "
Oct 30 22:41:31.104866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104703    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:31.104866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.104765    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:31.105223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.105199    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:31.105348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.105225    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.105348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.105249    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.105348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.105263    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b242cced-73d5-48cf-9819-84803277f9da-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.105348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.105277    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.106631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.106609    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:31.106922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.106894    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:31.107019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.106989    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "kube-api-access-c6fgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:31.107076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.107062    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy" pod "b242cced-73d5-48cf-9819-84803277f9da" (UID: "b242cced-73d5-48cf-9819-84803277f9da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:31.206511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.206472    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fgx\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-kube-api-access-c6fgx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.206511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.206509    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.206751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.206523    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b242cced-73d5-48cf-9819-84803277f9da-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.206751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.206537    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b242cced-73d5-48cf-9819-84803277f9da-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.206751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.206551    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b242cced-73d5-48cf-9819-84803277f9da-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:31.314108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.314079    5028 generic.go:356] "Generic (PLEG): container finished" podID="b242cced-73d5-48cf-9819-84803277f9da" containerID="f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc" exitCode=0
Oct 30 22:41:31.314248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.314130    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"
Oct 30 22:41:31.314248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.314157    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" event={"ID":"b242cced-73d5-48cf-9819-84803277f9da","Type":"ContainerDied","Data":"f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc"}
Oct 30 22:41:31.314248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.314216    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7" event={"ID":"b242cced-73d5-48cf-9819-84803277f9da","Type":"ContainerDied","Data":"3a11c5df6190935d3ae73ce747422ba9041e138754db56a0c0077aefba7eaf8a"}
Oct 30 22:41:31.314248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.314246    5028 scope.go:122] "RemoveContainer" containerID="f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc"
Oct 30 22:41:31.315679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.315656    5028 generic.go:356] "Generic (PLEG): container finished" podID="ad5232f8-2c29-407f-b212-917e30f59647" containerID="e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975" exitCode=0
Oct 30 22:41:31.315679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.315708    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"
Oct 30 22:41:31.315883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.315738    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" event={"ID":"ad5232f8-2c29-407f-b212-917e30f59647","Type":"ContainerDied","Data":"e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975"}
Oct 30 22:41:31.315883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.315789    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn" event={"ID":"ad5232f8-2c29-407f-b212-917e30f59647","Type":"ContainerDied","Data":"0db06371f3f437d3e46b3c3cf079572a991cd0f9d26aa8f810ce011c97283114"}
Oct 30 22:41:31.317446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.317422    5028 generic.go:356] "Generic (PLEG): container finished" podID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerID="a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427" exitCode=0
Oct 30 22:41:31.317565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.317478    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"
Oct 30 22:41:31.317641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.317613    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" event={"ID":"de55b3ff-0808-4f8d-95a2-8e8f046af5d7","Type":"ContainerDied","Data":"a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427"}
Oct 30 22:41:31.317726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.317659    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4" event={"ID":"de55b3ff-0808-4f8d-95a2-8e8f046af5d7","Type":"ContainerDied","Data":"4facf05295cee29bfceee6a7c37aee9fdd1d7295b105a371e4dda462b45b18e0"}
Oct 30 22:41:31.328085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.328065    5028 scope.go:122] "RemoveContainer" containerID="a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"
Oct 30 22:41:31.341536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.341512    5028 scope.go:122] "RemoveContainer" containerID="f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc"
Oct 30 22:41:31.342265 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:41:31.342236    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc\": container with ID starting with f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc not found: ID does not exist" containerID="f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc"
Oct 30 22:41:31.342371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.342277    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc"} err="failed to get container status \"f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc\": rpc error: code = NotFound desc = could not find container \"f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc\": container with ID starting with f385c874868299fce2576e2795eb466ffb72cd2fe171c8846b200ea2ab7a12bc not found: ID does not exist"
Oct 30 22:41:31.342371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.342306    5028 scope.go:122] "RemoveContainer" containerID="a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"
Oct 30 22:41:31.342646 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:41:31.342625    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908\": container with ID starting with a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908 not found: ID does not exist" containerID="a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"
Oct 30 22:41:31.342740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.342654    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908"} err="failed to get container status \"a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908\": rpc error: code = NotFound desc = could not find container \"a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908\": container with ID starting with a39d1b3e6e7ed1d2fed6d4aa053f94cfd45020aec834ea017a767c4d187e2908 not found: ID does not exist"
Oct 30 22:41:31.342740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.342678    5028 scope.go:122] "RemoveContainer" containerID="e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975"
Oct 30 22:41:31.353702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.353659    5028 scope.go:122] "RemoveContainer" containerID="e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975"
Oct 30 22:41:31.354032 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:41:31.354012    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975\": container with ID starting with e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975 not found: ID does not exist" containerID="e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975"
Oct 30 22:41:31.354107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.354044    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975"} err="failed to get container status \"e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975\": rpc error: code = NotFound desc = could not find container \"e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975\": container with ID starting with e5a5bcb17241c78571507062680fd4c65cdfda1544933d5355a4813feec2d975 not found: ID does not exist"
Oct 30 22:41:31.354107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.354064    5028 scope.go:122] "RemoveContainer" containerID="a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427"
Oct 30 22:41:31.364431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.364413    5028 scope.go:122] "RemoveContainer" containerID="a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427"
Oct 30 22:41:31.364757 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:41:31.364725    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427\": container with ID starting with a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427 not found: ID does not exist" containerID="a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427"
Oct 30 22:41:31.364817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.364758    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427"} err="failed to get container status \"a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427\": rpc error: code = NotFound desc = could not find container \"a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427\": container with ID starting with a3089adb0f284ea365a9e32247cca6d3ff2aafcb25e72aab868722e274a70427 not found: ID does not exist"
Oct 30 22:41:31.474834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.474789    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"]
Oct 30 22:41:31.480670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.480643    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-5dsv4"]
Oct 30 22:41:31.587766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.587659    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"]
Oct 30 22:41:31.591597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.591572    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qrmr7"]
Oct 30 22:41:31.695370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.695334    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"]
Oct 30 22:41:31.699454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.699425    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-j6skn"]
Oct 30 22:41:31.864022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:31.863939    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:33.293074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:33.293044    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5232f8-2c29-407f-b212-917e30f59647" path="/var/lib/kubelet/pods/ad5232f8-2c29-407f-b212-917e30f59647/volumes"
Oct 30 22:41:33.293552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:33.293538    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b242cced-73d5-48cf-9819-84803277f9da" path="/var/lib/kubelet/pods/b242cced-73d5-48cf-9819-84803277f9da/volumes"
Oct 30 22:41:33.294017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:33.294004    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" path="/var/lib/kubelet/pods/de55b3ff-0808-4f8d-95a2-8e8f046af5d7/volumes"
Oct 30 22:41:36.441833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:36.441792    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:36.745109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:36.745072    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:38.644621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:38.644585    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:40.982739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:40.982700    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:43.627058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:43.627029    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:45.587806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:45.587768    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:47.582110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582073    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"]
Oct 30 22:41:47.582749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582733    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy"
Oct 30 22:41:47.582749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582750    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy"
Oct 30 22:41:47.582893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582776    5028 cpu_manager.go:414] "containerMap: removing container" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy"
Oct 30 22:41:47.582893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582781    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy"
Oct 30 22:41:47.582893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582806    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy"
Oct 30 22:41:47.582893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582811    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy"
Oct 30 22:41:47.583012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582914    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy"
Oct 30 22:41:47.583012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582924    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5232f8-2c29-407f-b212-917e30f59647" containerName="istio-proxy"
Oct 30 22:41:47.583012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582935    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy"
Oct 30 22:41:47.583012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.582946    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="de55b3ff-0808-4f8d-95a2-8e8f046af5d7" containerName="istio-proxy"
Oct 30 22:41:47.586916 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.586897    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.630999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.630949    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb2th\" (UniqueName: \"kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.631173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.631012    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.631173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.631115    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.642257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.642226    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"]
Oct 30 22:41:47.683409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.683380    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:47.683635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.683612    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:47.683724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.683616    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:41:47.684342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.684323    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:47.732206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.732171    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb2th\" (UniqueName: \"kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.732419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.732213    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.732419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.732263    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.732944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.732915    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.735200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.735176    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.790851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.790815    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb2th\" (UniqueName: \"kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th\") pod \"maas-api-revoke-sub-cd0124df-65q46\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:47.901738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:47.901624    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:48.284123 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:41:48.284083    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ab0e5e_ba97_40d6_8455_271bae592cf4.slice/crio-b6e06709462bef437ad6290b5358a425413a47aef61709d75d45f58fffa390f3 WatchSource:0}: Error finding container b6e06709462bef437ad6290b5358a425413a47aef61709d75d45f58fffa390f3: Status 404 returned error can't find the container with id b6e06709462bef437ad6290b5358a425413a47aef61709d75d45f58fffa390f3
Oct 30 22:41:48.284281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:48.284203    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"]
Oct 30 22:41:48.478668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:48.478620    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46" event={"ID":"e4ab0e5e-ba97-40d6-8455-271bae592cf4","Type":"ContainerStarted","Data":"b6e06709462bef437ad6290b5358a425413a47aef61709d75d45f58fffa390f3"}
Oct 30 22:41:49.488943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:49.488898    5028 generic.go:356] "Generic (PLEG): container finished" podID="e4ab0e5e-ba97-40d6-8455-271bae592cf4" containerID="f186bfe394d7db5f2b3302fc63b13b6852bb6cc2360bb7a88a6f126d3c97a1d7" exitCode=0
Oct 30 22:41:49.489367 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:49.489012    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46" event={"ID":"e4ab0e5e-ba97-40d6-8455-271bae592cf4","Type":"ContainerDied","Data":"f186bfe394d7db5f2b3302fc63b13b6852bb6cc2360bb7a88a6f126d3c97a1d7"}
Oct 30 22:41:50.089638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.089596    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:50.664226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.664198    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:50.780351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.780272    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th\" (UniqueName: \"kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th\") pod \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") "
Oct 30 22:41:50.780649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.780625    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca\") pod \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") "
Oct 30 22:41:50.780839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.780821    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca\") pod \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\" (UID: \"e4ab0e5e-ba97-40d6-8455-271bae592cf4\") "
Oct 30 22:41:50.781817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.781786    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca" pod "e4ab0e5e-ba97-40d6-8455-271bae592cf4" (UID: "e4ab0e5e-ba97-40d6-8455-271bae592cf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:50.784422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.784394    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th" pod "e4ab0e5e-ba97-40d6-8455-271bae592cf4" (UID: "e4ab0e5e-ba97-40d6-8455-271bae592cf4"). InnerVolumeSpecName "kube-api-access-gb2th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:50.786153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.786122    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca" pod "e4ab0e5e-ba97-40d6-8455-271bae592cf4" (UID: "e4ab0e5e-ba97-40d6-8455-271bae592cf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:50.882422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.882388    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:50.882422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.882420    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb2th\" (UniqueName: \"kubernetes.io/projected/e4ab0e5e-ba97-40d6-8455-271bae592cf4-kube-api-access-gb2th\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:50.882635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:50.882434    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ab0e5e-ba97-40d6-8455-271bae592cf4-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:51.506587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:51.506558    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"
Oct 30 22:41:51.506793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:51.506601    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46" event={"ID":"e4ab0e5e-ba97-40d6-8455-271bae592cf4","Type":"ContainerDied","Data":"b6e06709462bef437ad6290b5358a425413a47aef61709d75d45f58fffa390f3"}
Oct 30 22:41:51.506793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:51.506659    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e06709462bef437ad6290b5358a425413a47aef61709d75d45f58fffa390f3"
Oct 30 22:41:54.516852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:54.516811    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.352929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.352891    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.437548 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:41:58.437506    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:41:58.815830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.815796    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"]
Oct 30 22:41:58.816491 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.816476    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy"
Oct 30 22:41:58.816545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.816493    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b242cced-73d5-48cf-9819-84803277f9da" containerName="istio-proxy"
Oct 30 22:41:58.816545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.816514    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e4ab0e5e-ba97-40d6-8455-271bae592cf4" containerName="revoke-keys"
Oct 30 22:41:58.816545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.816519    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ab0e5e-ba97-40d6-8455-271bae592cf4" containerName="revoke-keys"
Oct 30 22:41:58.816635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.816620    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ab0e5e-ba97-40d6-8455-271bae592cf4" containerName="revoke-keys"
Oct 30 22:41:58.820317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.820295    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:58.875383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.875351    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"]
Oct 30 22:41:58.917933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.917885    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:58.917933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.917889    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:58.917933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.917907    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:58.918208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.918101    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:41:58.986532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.986491    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:58.986756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.986589    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:58.986756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:58.986656    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq87p\" (UniqueName: \"kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.024166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.024131    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:59.088562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.088461    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.088562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.088527    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.088842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.088602    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq87p\" (UniqueName: \"kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.089441 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.089369    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.091141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.091119    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.148235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.148197    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq87p\" (UniqueName: \"kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p\") pod \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.436163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.436073    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:41:59.813481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:41:59.813445    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"]
Oct 30 22:41:59.814164 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:41:59.814136    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eaa7c72_9e43_4a16_a778_c1e2c05316ea.slice/crio-6f9faa19b338ebeccd6325d328e58b030de66aa6006cb788f4e7e4f6759a8cfc WatchSource:0}: Error finding container 6f9faa19b338ebeccd6325d328e58b030de66aa6006cb788f4e7e4f6759a8cfc: Status 404 returned error can't find the container with id 6f9faa19b338ebeccd6325d328e58b030de66aa6006cb788f4e7e4f6759a8cfc
Oct 30 22:42:00.486459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:00.486373    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:00.599053 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:00.599016    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv" event={"ID":"8eaa7c72-9e43-4a16-a778-c1e2c05316ea","Type":"ContainerStarted","Data":"f8747f2e71ed0b9df584e453e8b421e50e62c7d2b809b206dc30120b35ef24ed"}
Oct 30 22:42:00.599259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:00.599078    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv" event={"ID":"8eaa7c72-9e43-4a16-a778-c1e2c05316ea","Type":"ContainerStarted","Data":"6f9faa19b338ebeccd6325d328e58b030de66aa6006cb788f4e7e4f6759a8cfc"}
Oct 30 22:42:00.716673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:00.716612    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv" podStartSLOduration=2.7165974090000002 podStartE2EDuration="2.716597409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:00.714240886 +0000 UTC m=+2451.563753861" watchObservedRunningTime="2026-10-30 22:42:00.716597409 +0000 UTC m=+2451.566110382"
Oct 30 22:42:01.627245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:01.627209    5028 generic.go:356] "Generic (PLEG): container finished" podID="8eaa7c72-9e43-4a16-a778-c1e2c05316ea" containerID="f8747f2e71ed0b9df584e453e8b421e50e62c7d2b809b206dc30120b35ef24ed" exitCode=0
Oct 30 22:42:01.627860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:01.627834    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv" event={"ID":"8eaa7c72-9e43-4a16-a778-c1e2c05316ea","Type":"ContainerDied","Data":"f8747f2e71ed0b9df584e453e8b421e50e62c7d2b809b206dc30120b35ef24ed"}
Oct 30 22:42:02.789656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.789629    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:42:02.975958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.975919    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca\") pod \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") "
Oct 30 22:42:02.976149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.975978    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p\" (UniqueName: \"kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p\") pod \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") "
Oct 30 22:42:02.976149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.976031    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca\") pod \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\" (UID: \"8eaa7c72-9e43-4a16-a778-c1e2c05316ea\") "
Oct 30 22:42:02.976313 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.976286    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca" pod "8eaa7c72-9e43-4a16-a778-c1e2c05316ea" (UID: "8eaa7c72-9e43-4a16-a778-c1e2c05316ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:02.976593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.976574    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:02.978129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.978106    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca" pod "8eaa7c72-9e43-4a16-a778-c1e2c05316ea" (UID: "8eaa7c72-9e43-4a16-a778-c1e2c05316ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:02.978265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:02.978250    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p" pod "8eaa7c72-9e43-4a16-a778-c1e2c05316ea" (UID: "8eaa7c72-9e43-4a16-a778-c1e2c05316ea"). InnerVolumeSpecName "kube-api-access-jq87p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:03.077937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.077898    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq87p\" (UniqueName: \"kubernetes.io/projected/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-kube-api-access-jq87p\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:03.077937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.077935    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eaa7c72-9e43-4a16-a778-c1e2c05316ea-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:03.319432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.319392    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"]
Oct 30 22:42:03.330417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.330386    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"]
Oct 30 22:42:03.418191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.418156    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:03.645663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.644729    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv" event={"ID":"8eaa7c72-9e43-4a16-a778-c1e2c05316ea","Type":"ContainerDied","Data":"6f9faa19b338ebeccd6325d328e58b030de66aa6006cb788f4e7e4f6759a8cfc"}
Oct 30 22:42:03.645663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.644801    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:42:03.645663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.644823    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f9faa19b338ebeccd6325d328e58b030de66aa6006cb788f4e7e4f6759a8cfc"
Oct 30 22:42:03.716814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:03.716782    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:08.173262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:08.173223    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:10.223843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:10.223805    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"]
Oct 30 22:42:10.581150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:10.581066    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:10.754710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:10.754114    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:12.674420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:12.674387    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:13.923129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:13.923087    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:15.224865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:15.224820    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerName="payload-processing" containerID="cri-o://37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479" gracePeriod=25
Oct 30 22:42:16.625597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.625561    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"]
Oct 30 22:42:16.626343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.626321    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8eaa7c72-9e43-4a16-a778-c1e2c05316ea" containerName="revoke-keys"
Oct 30 22:42:16.626343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.626339    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eaa7c72-9e43-4a16-a778-c1e2c05316ea" containerName="revoke-keys"
Oct 30 22:42:16.626563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.626523    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eaa7c72-9e43-4a16-a778-c1e2c05316ea" containerName="revoke-keys"
Oct 30 22:42:16.630753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.630729    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.694365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.694334    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:42:16.698589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.698556    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"]
Oct 30 22:42:16.738483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.738434    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:16.738483 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.738466    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:16.738800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.738610    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:16.745980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.745948    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"]
Oct 30 22:42:16.750627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.750604    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.770724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.770678    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22268\" (UniqueName: \"kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.770914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.770750    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.770914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.770791    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.805551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.805508    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"]
Oct 30 22:42:16.871968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.871925    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.871968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.871973    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.872273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.872156    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22268\" (UniqueName: \"kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.872273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.872195    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb72\" (UniqueName: \"kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.872273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.872251    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.872448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.872368    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.872969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.872947    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.874828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.874797    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.934863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.934778    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22268\" (UniqueName: \"kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268\") pod \"maas-api-revoke-sub-49c7a229-q6qbp\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.945500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.945333    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:16.974489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.974455    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.974666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.974508    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.974666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.974588    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb72\" (UniqueName: \"kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.975536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.975509    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:16.977509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:16.977486    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:17.057844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.057727    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb72\" (UniqueName: \"kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72\") pod \"maas-api-revoke-sub-c10816ca-m8qc9\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:17.065699 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.065408    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:17.334799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.334768    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"]
Oct 30 22:42:17.335455 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:42:17.335430    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd5384_7330_4b73_af79_21af15129ca6.slice/crio-843d18c7c33ec8247ee7f699efb01aed4a5ef0a08ddbe9d5a6e77c1e3b8c1a7d WatchSource:0}: Error finding container 843d18c7c33ec8247ee7f699efb01aed4a5ef0a08ddbe9d5a6e77c1e3b8c1a7d: Status 404 returned error can't find the container with id 843d18c7c33ec8247ee7f699efb01aed4a5ef0a08ddbe9d5a6e77c1e3b8c1a7d
Oct 30 22:42:17.482113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.482080    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"]
Oct 30 22:42:17.494432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.494401    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:17.520013 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:42:17.519957    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba08d36_492d_4b55_aca5_52d77488e2ec.slice/crio-aa6055d37a9afe5d9c538d6e247c02d1eb5fcd75c3a656309006c5e9980dc24c WatchSource:0}: Error finding container aa6055d37a9afe5d9c538d6e247c02d1eb5fcd75c3a656309006c5e9980dc24c: Status 404 returned error can't find the container with id aa6055d37a9afe5d9c538d6e247c02d1eb5fcd75c3a656309006c5e9980dc24c
Oct 30 22:42:17.820124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.820084    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9" event={"ID":"5ba08d36-492d-4b55-aca5-52d77488e2ec","Type":"ContainerStarted","Data":"aa6055d37a9afe5d9c538d6e247c02d1eb5fcd75c3a656309006c5e9980dc24c"}
Oct 30 22:42:17.822450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:17.822416    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp" event={"ID":"26fd5384-7330-4b73-af79-21af15129ca6","Type":"ContainerStarted","Data":"843d18c7c33ec8247ee7f699efb01aed4a5ef0a08ddbe9d5a6e77c1e3b8c1a7d"}
Oct 30 22:42:18.215415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:18.215382    5028 scope.go:122] "RemoveContainer" containerID="82952cc884f8883c441f4a2989f60b78e864e1c72f4b80603bdd2f6413737375"
Oct 30 22:42:18.232242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:18.232212    5028 scope.go:122] "RemoveContainer" containerID="d9570317dad0c7ad5b0c25cdd95ab15e9cc41d73753d6e2fed60fb7f2bf4bd4d"
Oct 30 22:42:18.310572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:18.310437    5028 scope.go:122] "RemoveContainer" containerID="4386c5826ef033f74872f7147c0af08d87a01f50f2190ec3fe90f32a54fdedf9"
Oct 30 22:42:18.349086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:18.348894    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" containerID="cri-o://10d765efc36a606ed01c60db3042b27e110e9d47e0ad30b61111ac08b4ee42d5" gracePeriod=45
Oct 30 22:42:18.369157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:18.369112    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" containerID="cri-o://e164b0aea4856c63ea6774a61f63024c1562d0aa4ad114fea7714d26bf7f87c6" gracePeriod=45
Oct 30 22:42:18.526621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:18.523096    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": dial tcp 10.132.0.113:8000: connect: connection refused"
Oct 30 22:42:19.022995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.012619    5028 generic.go:356] "Generic (PLEG): container finished" podID="5ba08d36-492d-4b55-aca5-52d77488e2ec" containerID="0aa4c29139345efbe2c4e33d1be8e5cab21f56ad11cc36574de35b86465d2f6a" exitCode=0
Oct 30 22:42:19.022995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.012802    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9" event={"ID":"5ba08d36-492d-4b55-aca5-52d77488e2ec","Type":"ContainerDied","Data":"0aa4c29139345efbe2c4e33d1be8e5cab21f56ad11cc36574de35b86465d2f6a"}
Oct 30 22:42:19.022995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.020990    5028 generic.go:356] "Generic (PLEG): container finished" podID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerID="10d765efc36a606ed01c60db3042b27e110e9d47e0ad30b61111ac08b4ee42d5" exitCode=0
Oct 30 22:42:19.022995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.021119    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" event={"ID":"c2b354fa-5137-4c50-b37e-c77ed3249c95","Type":"ContainerDied","Data":"10d765efc36a606ed01c60db3042b27e110e9d47e0ad30b61111ac08b4ee42d5"}
Oct 30 22:42:19.023659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.023250    5028 generic.go:356] "Generic (PLEG): container finished" podID="26fd5384-7330-4b73-af79-21af15129ca6" containerID="4e5c9ed3e50d7f9f23eb428eabb10e09fa80380faeefcdb762bba23569652fa2" exitCode=0
Oct 30 22:42:19.023659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.023377    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp" event={"ID":"26fd5384-7330-4b73-af79-21af15129ca6","Type":"ContainerDied","Data":"4e5c9ed3e50d7f9f23eb428eabb10e09fa80380faeefcdb762bba23569652fa2"}
Oct 30 22:42:19.035779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.025634    5028 generic.go:356] "Generic (PLEG): container finished" podID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerID="e164b0aea4856c63ea6774a61f63024c1562d0aa4ad114fea7714d26bf7f87c6" exitCode=0
Oct 30 22:42:19.035779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.025795    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" event={"ID":"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789","Type":"ContainerDied","Data":"e164b0aea4856c63ea6774a61f63024c1562d0aa4ad114fea7714d26bf7f87c6"}
Oct 30 22:42:19.114998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.114679    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:42:19.132865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.128159    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:42:19.192309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.192020    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir\") pod \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") "
Oct 30 22:42:19.192309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.192081    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home\") pod \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") "
Oct 30 22:42:19.192309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.192138    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm\") pod \"c2b354fa-5137-4c50-b37e-c77ed3249c95\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") "
Oct 30 22:42:19.192309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.192165    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm\") pod \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") "
Oct 30 22:42:19.192309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.192194    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2\" (UniqueName: \"kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2\") pod \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") "
Oct 30 22:42:19.192309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.192234    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache\") pod \"c2b354fa-5137-4c50-b37e-c77ed3249c95\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") "
Oct 30 22:42:19.194422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.194364    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache" pod "c2b354fa-5137-4c50-b37e-c77ed3249c95" (UID: "c2b354fa-5137-4c50-b37e-c77ed3249c95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.195101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.195068    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir" pod "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" (UID: "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.195308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.195269    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home" pod "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" (UID: "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.196038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.196011    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm" pod "c2b354fa-5137-4c50-b37e-c77ed3249c95" (UID: "c2b354fa-5137-4c50-b37e-c77ed3249c95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.209480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.209437    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2" pod "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" (UID: "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789"). InnerVolumeSpecName "kube-api-access-nwpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:19.210061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.210031    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs\") pod \"c2b354fa-5137-4c50-b37e-c77ed3249c95\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") "
Oct 30 22:42:19.210199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.210145    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc\" (UniqueName: \"kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc\") pod \"c2b354fa-5137-4c50-b37e-c77ed3249c95\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") "
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.210319    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir\") pod \"c2b354fa-5137-4c50-b37e-c77ed3249c95\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") "
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.211037    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir" pod "c2b354fa-5137-4c50-b37e-c77ed3249c95" (UID: "c2b354fa-5137-4c50-b37e-c77ed3249c95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.213026    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm" pod "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" (UID: "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.218337    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home\") pod \"c2b354fa-5137-4c50-b37e-c77ed3249c95\" (UID: \"c2b354fa-5137-4c50-b37e-c77ed3249c95\") "
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.218383    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs\") pod \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") "
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.218446    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache\") pod \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\" (UID: \"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789\") "
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219203    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219231    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219245    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219257    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219269    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219283    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwpk2\" (UniqueName: \"kubernetes.io/projected/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-kube-api-access-nwpk2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219299    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219541    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache" pod "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" (UID: "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.230733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.219790    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home" pod "c2b354fa-5137-4c50-b37e-c77ed3249c95" (UID: "c2b354fa-5137-4c50-b37e-c77ed3249c95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:19.231967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.231914    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs" pod "c2b354fa-5137-4c50-b37e-c77ed3249c95" (UID: "c2b354fa-5137-4c50-b37e-c77ed3249c95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:19.232723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.232671    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc" pod "c2b354fa-5137-4c50-b37e-c77ed3249c95" (UID: "c2b354fa-5137-4c50-b37e-c77ed3249c95"). InnerVolumeSpecName "kube-api-access-jhdvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:19.314466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.314359    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs" pod "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" (UID: "3eb5ee88-1511-42b1-8f5f-2f5eed4ee789"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:19.324984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.324914    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b354fa-5137-4c50-b37e-c77ed3249c95-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.376742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.376407    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhdvc\" (UniqueName: \"kubernetes.io/projected/c2b354fa-5137-4c50-b37e-c77ed3249c95-kube-api-access-jhdvc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.376742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.376446    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b354fa-5137-4c50-b37e-c77ed3249c95-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.376742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.376462    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.376742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.376478    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:19.976192 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:19.975141    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:42:20.037632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.037591    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck" event={"ID":"c2b354fa-5137-4c50-b37e-c77ed3249c95","Type":"ContainerDied","Data":"5a2eb30d967cb2d7fd37b920c4f3d5833c294ec79764004cc10a33ec950bb0e4"}
Oct 30 22:42:20.038156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.037647    5028 scope.go:122] "RemoveContainer" containerID="10d765efc36a606ed01c60db3042b27e110e9d47e0ad30b61111ac08b4ee42d5"
Oct 30 22:42:20.038156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.037837    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"
Oct 30 22:42:20.042437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.042411    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"
Oct 30 22:42:20.043719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.042965    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m" event={"ID":"3eb5ee88-1511-42b1-8f5f-2f5eed4ee789","Type":"ContainerDied","Data":"605ada37cf8426d4399357a795c4c5f6e338cfea93f35ba1fb742f6c928d6775"}
Oct 30 22:42:20.069547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.067527    5028 scope.go:122] "RemoveContainer" containerID="e164b0aea4856c63ea6774a61f63024c1562d0aa4ad114fea7714d26bf7f87c6"
Oct 30 22:42:20.291838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.291734    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"]
Oct 30 22:42:20.296918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.296834    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-2-w6-be17e7-kserve-68c5df7c89-ztfck"]
Oct 30 22:42:20.384677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.384595    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"]
Oct 30 22:42:20.414394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.414337    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/premium-w6-be17e7-kserve-77f4b7f5f-4xc4m"]
Oct 30 22:42:20.415404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.415373    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:20.511635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.511570    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca\") pod \"26fd5384-7330-4b73-af79-21af15129ca6\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") "
Oct 30 22:42:20.511903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.511775    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca\") pod \"26fd5384-7330-4b73-af79-21af15129ca6\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") "
Oct 30 22:42:20.512010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.511983    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268\" (UniqueName: \"kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268\") pod \"26fd5384-7330-4b73-af79-21af15129ca6\" (UID: \"26fd5384-7330-4b73-af79-21af15129ca6\") "
Oct 30 22:42:20.512127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.512109    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:20.512975 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.512949    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca" pod "26fd5384-7330-4b73-af79-21af15129ca6" (UID: "26fd5384-7330-4b73-af79-21af15129ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:20.515208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.515180    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca" pod "26fd5384-7330-4b73-af79-21af15129ca6" (UID: "26fd5384-7330-4b73-af79-21af15129ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:20.516075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.516048    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268" pod "26fd5384-7330-4b73-af79-21af15129ca6" (UID: "26fd5384-7330-4b73-af79-21af15129ca6"). InnerVolumeSpecName "kube-api-access-22268". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:20.598190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.598005    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:20.615101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.613825    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72\" (UniqueName: \"kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72\") pod \"5ba08d36-492d-4b55-aca5-52d77488e2ec\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") "
Oct 30 22:42:20.615101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.614006    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca\") pod \"5ba08d36-492d-4b55-aca5-52d77488e2ec\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") "
Oct 30 22:42:20.615101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.614064    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca\") pod \"5ba08d36-492d-4b55-aca5-52d77488e2ec\" (UID: \"5ba08d36-492d-4b55-aca5-52d77488e2ec\") "
Oct 30 22:42:20.615101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.614676    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22268\" (UniqueName: \"kubernetes.io/projected/26fd5384-7330-4b73-af79-21af15129ca6-kube-api-access-22268\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:20.615101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.614716    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:20.615101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.614732    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fd5384-7330-4b73-af79-21af15129ca6-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:20.616447 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.615879    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca" pod "5ba08d36-492d-4b55-aca5-52d77488e2ec" (UID: "5ba08d36-492d-4b55-aca5-52d77488e2ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:20.695042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.694999    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72" pod "5ba08d36-492d-4b55-aca5-52d77488e2ec" (UID: "5ba08d36-492d-4b55-aca5-52d77488e2ec"). InnerVolumeSpecName "kube-api-access-cfb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:20.695909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.695865    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca" pod "5ba08d36-492d-4b55-aca5-52d77488e2ec" (UID: "5ba08d36-492d-4b55-aca5-52d77488e2ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:20.717075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.716097    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfb72\" (UniqueName: \"kubernetes.io/projected/5ba08d36-492d-4b55-aca5-52d77488e2ec-kube-api-access-cfb72\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:20.717075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.716137    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:20.717075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:20.716155    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba08d36-492d-4b55-aca5-52d77488e2ec-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:21.053832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.053803    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"
Oct 30 22:42:21.054320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.053832    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp" event={"ID":"26fd5384-7330-4b73-af79-21af15129ca6","Type":"ContainerDied","Data":"843d18c7c33ec8247ee7f699efb01aed4a5ef0a08ddbe9d5a6e77c1e3b8c1a7d"}
Oct 30 22:42:21.054320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.053897    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843d18c7c33ec8247ee7f699efb01aed4a5ef0a08ddbe9d5a6e77c1e3b8c1a7d"
Oct 30 22:42:21.056845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.056811    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9" event={"ID":"5ba08d36-492d-4b55-aca5-52d77488e2ec","Type":"ContainerDied","Data":"aa6055d37a9afe5d9c538d6e247c02d1eb5fcd75c3a656309006c5e9980dc24c"}
Oct 30 22:42:21.057008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.056866    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6055d37a9afe5d9c538d6e247c02d1eb5fcd75c3a656309006c5e9980dc24c"
Oct 30 22:42:21.057008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.056829    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"
Oct 30 22:42:21.299241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.299161    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" path="/var/lib/kubelet/pods/3eb5ee88-1511-42b1-8f5f-2f5eed4ee789/volumes"
Oct 30 22:42:21.299633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:21.299618    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" path="/var/lib/kubelet/pods/c2b354fa-5137-4c50-b37e-c77ed3249c95/volumes"
Oct 30 22:42:22.018363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.018320    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:22.433297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.433210    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:22.968339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.968303    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"]
Oct 30 22:42:22.969033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969015    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5ba08d36-492d-4b55-aca5-52d77488e2ec" containerName="revoke-keys"
Oct 30 22:42:22.969033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969033    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba08d36-492d-4b55-aca5-52d77488e2ec" containerName="revoke-keys"
Oct 30 22:42:22.969161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969049    5028 cpu_manager.go:414] "containerMap: removing container" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main"
Oct 30 22:42:22.969161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969054    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main"
Oct 30 22:42:22.969161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969065    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main"
Oct 30 22:42:22.969161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969070    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main"
Oct 30 22:42:22.969161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969091    5028 cpu_manager.go:414] "containerMap: removing container" podUID="26fd5384-7330-4b73-af79-21af15129ca6" containerName="revoke-keys"
Oct 30 22:42:22.969161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969097    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fd5384-7330-4b73-af79-21af15129ca6" containerName="revoke-keys"
Oct 30 22:42:22.969358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969184    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fd5384-7330-4b73-af79-21af15129ca6" containerName="revoke-keys"
Oct 30 22:42:22.969358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969200    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba08d36-492d-4b55-aca5-52d77488e2ec" containerName="revoke-keys"
Oct 30 22:42:22.969358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969209    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb5ee88-1511-42b1-8f5f-2f5eed4ee789" containerName="main"
Oct 30 22:42:22.969358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.969217    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b354fa-5137-4c50-b37e-c77ed3249c95" containerName="main"
Oct 30 22:42:22.974207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:22.974186    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.027891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.027851    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"]
Oct 30 22:42:23.069960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.069919    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:23.070119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.069973    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:23.071242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.071221    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:23.071397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.071298    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:42:23.145074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.145036    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.145288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.145209    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gs2m\" (UniqueName: \"kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.145288 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.145249    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.246978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.246874    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.247181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.247021    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gs2m\" (UniqueName: \"kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.247181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.247053    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.247567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.247543    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.249479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.249460    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.306066 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.306029    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gs2m\" (UniqueName: \"kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m\") pod \"maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.588382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.588293    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:23.971332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:23.971192    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"]
Oct 30 22:42:24.084351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:24.084300    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk" event={"ID":"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb","Type":"ContainerStarted","Data":"90d6b1a7b2f55ab0a85b7fe7f45bbfcfac4c6309c5fb991272e774b7dab906da"}
Oct 30 22:42:25.094498 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:25.094467    5028 generic.go:356] "Generic (PLEG): container finished" podID="6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" containerID="fe4a41b5c9153e005c03cac747dba67868cbe9489e1f08b8bbe95f10ae3b62d7" exitCode=0
Oct 30 22:42:25.094927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:25.094551    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk" event={"ID":"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb","Type":"ContainerDied","Data":"fe4a41b5c9153e005c03cac747dba67868cbe9489e1f08b8bbe95f10ae3b62d7"}
Oct 30 22:42:26.263364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.263334    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:26.389062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.388982    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m\" (UniqueName: \"kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m\") pod \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") "
Oct 30 22:42:26.389062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.389031    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca\") pod \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") "
Oct 30 22:42:26.389312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.389159    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca\") pod \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\" (UID: \"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb\") "
Oct 30 22:42:26.390903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.390841    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca" pod "6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" (UID: "6fce9bdc-295b-445e-a733-7c4bd5d5b4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:26.391806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.391779    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m" pod "6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" (UID: "6fce9bdc-295b-445e-a733-7c4bd5d5b4fb"). InnerVolumeSpecName "kube-api-access-9gs2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:26.392055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.392038    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca" pod "6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" (UID: "6fce9bdc-295b-445e-a733-7c4bd5d5b4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:26.491326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.491289    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gs2m\" (UniqueName: \"kubernetes.io/projected/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-kube-api-access-9gs2m\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.491326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.491322    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:26.491326 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:26.491332    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:27.113871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:27.113838    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"
Oct 30 22:42:27.114063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:27.113844    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk" event={"ID":"6fce9bdc-295b-445e-a733-7c4bd5d5b4fb","Type":"ContainerDied","Data":"90d6b1a7b2f55ab0a85b7fe7f45bbfcfac4c6309c5fb991272e774b7dab906da"}
Oct 30 22:42:27.114063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:27.113969    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d6b1a7b2f55ab0a85b7fe7f45bbfcfac4c6309c5fb991272e774b7dab906da"
Oct 30 22:42:27.177708 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:27.177658    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.438184 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:42:28.438146    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:42:31.744205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:31.744145    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:32.017217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:32.017124    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:42:34.372260 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:34.372210    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"]
Oct 30 22:42:34.474969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:34.474928    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"]
Oct 30 22:42:34.728441 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:34.728406    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:34.894519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:34.892617    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:35.512673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.512645    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:42:35.602220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.602185    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume\") pod \"9d0f8411-c36e-4301-aaa7-af10dacadb51\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") "
Oct 30 22:42:35.602432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.602296    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl\" (UniqueName: \"kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl\") pod \"9d0f8411-c36e-4301-aaa7-af10dacadb51\" (UID: \"9d0f8411-c36e-4301-aaa7-af10dacadb51\") "
Oct 30 22:42:35.602671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.602643    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume" pod "9d0f8411-c36e-4301-aaa7-af10dacadb51" (UID: "9d0f8411-c36e-4301-aaa7-af10dacadb51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:35.602812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.602796    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0f8411-c36e-4301-aaa7-af10dacadb51-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.604495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.604474    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl" pod "9d0f8411-c36e-4301-aaa7-af10dacadb51" (UID: "9d0f8411-c36e-4301-aaa7-af10dacadb51"). InnerVolumeSpecName "kube-api-access-6klpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:35.704358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:35.704320    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6klpl\" (UniqueName: \"kubernetes.io/projected/9d0f8411-c36e-4301-aaa7-af10dacadb51-kube-api-access-6klpl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:36.190799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.190765    5028 generic.go:356] "Generic (PLEG): container finished" podID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerID="37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479" exitCode=0
Oct 30 22:42:36.191019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.190847    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"
Oct 30 22:42:36.191019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.190853    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" event={"ID":"9d0f8411-c36e-4301-aaa7-af10dacadb51","Type":"ContainerDied","Data":"37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479"}
Oct 30 22:42:36.191019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.190938    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v" event={"ID":"9d0f8411-c36e-4301-aaa7-af10dacadb51","Type":"ContainerDied","Data":"a18dec83b4c9db9cfe209e3243bd2889a3c901880231ffdea020948781401cdb"}
Oct 30 22:42:36.191019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.190961    5028 scope.go:122] "RemoveContainer" containerID="37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479"
Oct 30 22:42:36.202176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.202156    5028 scope.go:122] "RemoveContainer" containerID="37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479"
Oct 30 22:42:36.202468 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:42:36.202447    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479\": container with ID starting with 37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479 not found: ID does not exist" containerID="37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479"
Oct 30 22:42:36.202513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.202478    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479"} err="failed to get container status \"37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479\": rpc error: code = NotFound desc = could not find container \"37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479\": container with ID starting with 37d60712435bef3651d73790fbbb45ecd9b5da6d730532f39084aa9cd5341479 not found: ID does not exist"
Oct 30 22:42:36.237419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.237388    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:36.351877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.351841    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"]
Oct 30 22:42:36.400666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:36.400628    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-be17e7-546f498b5c-f5b7v"]
Oct 30 22:42:37.292122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:37.292087    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" path="/var/lib/kubelet/pods/9d0f8411-c36e-4301-aaa7-af10dacadb51/volumes"
Oct 30 22:42:39.373308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.373253    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x" podUID="638522ab-0274-4468-9e3f-ed8e8219a080" containerName="payload-processing" containerID="cri-o://5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09" gracePeriod=25
Oct 30 22:42:39.476118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.476075    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj" podUID="8effebf9-048b-4365-80ed-f9cb50d2d6b9" containerName="payload-pre-processing" containerID="cri-o://178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8" gracePeriod=25
Oct 30 22:42:39.653088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.651589    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:39.731788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.731759    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:42:39.815511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.815471    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:39.863251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.863212    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf\" (UniqueName: \"kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf\") pod \"638522ab-0274-4468-9e3f-ed8e8219a080\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") "
Oct 30 22:42:39.863494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.863471    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume\") pod \"638522ab-0274-4468-9e3f-ed8e8219a080\" (UID: \"638522ab-0274-4468-9e3f-ed8e8219a080\") "
Oct 30 22:42:39.864369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.864341    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume" pod "638522ab-0274-4468-9e3f-ed8e8219a080" (UID: "638522ab-0274-4468-9e3f-ed8e8219a080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:39.866870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.866838    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf" pod "638522ab-0274-4468-9e3f-ed8e8219a080" (UID: "638522ab-0274-4468-9e3f-ed8e8219a080"). InnerVolumeSpecName "kube-api-access-5hngf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:39.875218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.875183    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:42:39.965016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.964986    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls\" (UniqueName: \"kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls\") pod \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") "
Oct 30 22:42:39.965195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.965030    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume\") pod \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\" (UID: \"8effebf9-048b-4365-80ed-f9cb50d2d6b9\") "
Oct 30 22:42:39.965440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.965417    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume" pod "8effebf9-048b-4365-80ed-f9cb50d2d6b9" (UID: "8effebf9-048b-4365-80ed-f9cb50d2d6b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:39.965721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.965674    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8effebf9-048b-4365-80ed-f9cb50d2d6b9-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:39.965767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.965725    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/638522ab-0274-4468-9e3f-ed8e8219a080-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:39.965767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.965736    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hngf\" (UniqueName: \"kubernetes.io/projected/638522ab-0274-4468-9e3f-ed8e8219a080-kube-api-access-5hngf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:39.967246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:39.967223    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls" pod "8effebf9-048b-4365-80ed-f9cb50d2d6b9" (UID: "8effebf9-048b-4365-80ed-f9cb50d2d6b9"). InnerVolumeSpecName "kube-api-access-kmlls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:40.067115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.067074    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmlls\" (UniqueName: \"kubernetes.io/projected/8effebf9-048b-4365-80ed-f9cb50d2d6b9-kube-api-access-kmlls\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:40.259972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.258760    5028 generic.go:356] "Generic (PLEG): container finished" podID="8effebf9-048b-4365-80ed-f9cb50d2d6b9" containerID="178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8" exitCode=0
Oct 30 22:42:40.259972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.259057    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj" event={"ID":"8effebf9-048b-4365-80ed-f9cb50d2d6b9","Type":"ContainerDied","Data":"178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8"}
Oct 30 22:42:40.259972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.259161    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj" event={"ID":"8effebf9-048b-4365-80ed-f9cb50d2d6b9","Type":"ContainerDied","Data":"72c66b2760dff2df93346edda971fed578b9a35c313f5844162bd74733b08686"}
Oct 30 22:42:40.259972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.259193    5028 scope.go:122] "RemoveContainer" containerID="178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8"
Oct 30 22:42:40.259972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.259363    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"
Oct 30 22:42:40.264374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.264340    5028 generic.go:356] "Generic (PLEG): container finished" podID="638522ab-0274-4468-9e3f-ed8e8219a080" containerID="5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09" exitCode=0
Oct 30 22:42:40.264709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.264664    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x" event={"ID":"638522ab-0274-4468-9e3f-ed8e8219a080","Type":"ContainerDied","Data":"5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09"}
Oct 30 22:42:40.264917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.264890    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x" event={"ID":"638522ab-0274-4468-9e3f-ed8e8219a080","Type":"ContainerDied","Data":"5c3a1a0d3f8225af29b9cf7b424ece38c66cabc3d0a4e523f3a2612272fe8b57"}
Oct 30 22:42:40.265155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.265139    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"
Oct 30 22:42:40.295616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.295550    5028 scope.go:122] "RemoveContainer" containerID="178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8"
Oct 30 22:42:40.296274 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:42:40.296208    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8\": container with ID starting with 178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8 not found: ID does not exist" containerID="178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8"
Oct 30 22:42:40.296414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.296286    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8"} err="failed to get container status \"178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8\": rpc error: code = NotFound desc = could not find container \"178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8\": container with ID starting with 178efa1066278cc5108bdf5e65d3774e1fb7522aad38eae95277b12aabcbc1f8 not found: ID does not exist"
Oct 30 22:42:40.296414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.296317    5028 scope.go:122] "RemoveContainer" containerID="5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09"
Oct 30 22:42:40.309387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.309357    5028 scope.go:122] "RemoveContainer" containerID="5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09"
Oct 30 22:42:40.309756 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:42:40.309729    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09\": container with ID starting with 5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09 not found: ID does not exist" containerID="5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09"
Oct 30 22:42:40.309862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.309764    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09"} err="failed to get container status \"5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09\": rpc error: code = NotFound desc = could not find container \"5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09\": container with ID starting with 5578bf69d49ac61bb1bcbd2baa1010747770ab84fe29c56c3d7bea8a8cda5a09 not found: ID does not exist"
Oct 30 22:42:40.482091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.482045    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"]
Oct 30 22:42:40.488601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.488569    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-771ff9-76cdd87b7f-qz5hj"]
Oct 30 22:42:40.594234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.594137    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"]
Oct 30 22:42:40.599099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.599068    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-771ff9-65d554b547-ct57x"]
Oct 30 22:42:40.794820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:40.794785    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:41.293104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:41.293068    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638522ab-0274-4468-9e3f-ed8e8219a080" path="/var/lib/kubelet/pods/638522ab-0274-4468-9e3f-ed8e8219a080/volumes"
Oct 30 22:42:41.293740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:41.293681    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8effebf9-048b-4365-80ed-f9cb50d2d6b9" path="/var/lib/kubelet/pods/8effebf9-048b-4365-80ed-f9cb50d2d6b9/volumes"
Oct 30 22:42:42.549089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:42.549048    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:44.242886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:44.242843    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:44.460861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:44.460822    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:45.382072 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:45.382036    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:48.402236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:48.402200    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:48.552934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:48.552902    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:49.849917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:49.849884    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:54.376702 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:54.376645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:58.437202 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:42:58.437165    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:42:58.826948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:42:58.826910    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:03.305718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:03.305669    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:07.222798 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:07.222755    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:08.647433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:08.647393    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:08.852825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:08.852786    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:09.004101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:09.004051    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:09.245812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:09.245773    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"]
Oct 30 22:43:09.293431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:09.293359    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"]
Oct 30 22:43:09.336824 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:43:09.336781    5028 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv\" is forbidden: User \"system:node:ip-10-0-129-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-125.us-west-2.compute.internal' and this object" podUID="8eaa7c72-9e43-4a16-a778-c1e2c05316ea" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-be17e7-pwvxv"
Oct 30 22:43:09.686430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:09.686349    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:09.959974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:09.959941    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:11.291743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:11.291682    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eaa7c72-9e43-4a16-a778-c1e2c05316ea" path="/var/lib/kubelet/pods/8eaa7c72-9e43-4a16-a778-c1e2c05316ea/volumes"
Oct 30 22:43:13.649881 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:13.649846    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:14.277448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:14.277408    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:16.078983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:16.078947    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:16.232283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:16.232219    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:16.529726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:16.529657    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:18.246969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:18.246935    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:18.622275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:18.622191    5028 scope.go:122] "RemoveContainer" containerID="1c4074393f876ebba62e1a86dcd83f1f9c20be9a6e9d7f66870f79878095c38d"
Oct 30 22:43:22.742103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:22.742065    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:26.676031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:26.675989    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:26.828071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:26.828038    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.245430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:27.245389    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:28.436856 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:43:28.436809    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:43:31.759812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:31.759776    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:36.260516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:36.260469    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:40.767955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:40.767919    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:45.250746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:45.250708    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:49.209760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:49.209673    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:50.630778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:50.630727    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:50.841537 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:50.841500    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:50.997934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:50.997892    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:51.092968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:51.092929    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"]
Oct 30 22:43:51.096717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:51.096674    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771ff9-lvbzk"]
Oct 30 22:43:51.291561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:51.291482    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" path="/var/lib/kubelet/pods/6fce9bdc-295b-445e-a733-7c4bd5d5b4fb/volumes"
Oct 30 22:43:51.551989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:51.551895    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:55.643135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:55.643095    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:57.870837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:57.870796    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:58.031707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:58.031646    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:58.191084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:58.190998    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"]
Oct 30 22:43:58.191249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:58.191214    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" containerID="cri-o://af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf" gracePeriod=30
Oct 30 22:43:58.194872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:58.194830    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:58.436509 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:43:58.436474    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:43:59.022010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.021976    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"]
Oct 30 22:43:59.022612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022596    5028 cpu_manager.go:414] "containerMap: removing container" podUID="638522ab-0274-4468-9e3f-ed8e8219a080" containerName="payload-processing"
Oct 30 22:43:59.022659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022613    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638522ab-0274-4468-9e3f-ed8e8219a080" containerName="payload-processing"
Oct 30 22:43:59.022659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022635    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8effebf9-048b-4365-80ed-f9cb50d2d6b9" containerName="payload-pre-processing"
Oct 30 22:43:59.022659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022640    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8effebf9-048b-4365-80ed-f9cb50d2d6b9" containerName="payload-pre-processing"
Oct 30 22:43:59.022659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022657    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerName="payload-processing"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022662    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerName="payload-processing"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022669    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" containerName="revoke-keys"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022675    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" containerName="revoke-keys"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022785    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8effebf9-048b-4365-80ed-f9cb50d2d6b9" containerName="payload-pre-processing"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022796    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0f8411-c36e-4301-aaa7-af10dacadb51" containerName="payload-processing"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022802    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fce9bdc-295b-445e-a733-7c4bd5d5b4fb" containerName="revoke-keys"
Oct 30 22:43:59.022849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.022814    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="638522ab-0274-4468-9e3f-ed8e8219a080" containerName="payload-processing"
Oct 30 22:43:59.027500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.027483    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.081421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.081388    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"]
Oct 30 22:43:59.119159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.119115    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.119159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.119160    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.119516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.119381    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctr7\" (UniqueName: \"kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.123039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.123017    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:59.123163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.123108    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:43:59.123163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.123125    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:59.123163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.123108    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:59.220936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.220893    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.220936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.220936    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.221213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.221124    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctr7\" (UniqueName: \"kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.221736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.221708    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.223433 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.223413    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.280271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.280192    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctr7\" (UniqueName: \"kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7\") pod \"maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.341951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.341905    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:43:59.723274 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:43:59.723244    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4f38ca_4642_4454_aa9d_24f7641b6874.slice/crio-41c45dc7cf1854237479a3805c454ddb9b68f7199e3262ae872e4b11df65e764 WatchSource:0}: Error finding container 41c45dc7cf1854237479a3805c454ddb9b68f7199e3262ae872e4b11df65e764: Status 404 returned error can't find the container with id 41c45dc7cf1854237479a3805c454ddb9b68f7199e3262ae872e4b11df65e764
Oct 30 22:43:59.723431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:43:59.723412    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"]
Oct 30 22:44:00.108634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:00.108592    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh" event={"ID":"ef4f38ca-4642-4454-aa9d-24f7641b6874","Type":"ContainerStarted","Data":"41c45dc7cf1854237479a3805c454ddb9b68f7199e3262ae872e4b11df65e764"}
Oct 30 22:44:00.429448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:00.429365    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:01.064166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:01.064077    5028 patch_prober.go:29] interesting pod/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:44:01.064166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:01.064140    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:44:01.117990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:01.117956    5028 generic.go:356] "Generic (PLEG): container finished" podID="ef4f38ca-4642-4454-aa9d-24f7641b6874" containerID="80b8d23b67038ec38be0a96b54a8bab6a2fe1104a6c2a327309874c9d6dc7ced" exitCode=0
Oct 30 22:44:01.118382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:01.118044    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh" event={"ID":"ef4f38ca-4642-4454-aa9d-24f7641b6874","Type":"ContainerDied","Data":"80b8d23b67038ec38be0a96b54a8bab6a2fe1104a6c2a327309874c9d6dc7ced"}
Oct 30 22:44:02.293146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.293117    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:44:02.360141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.360102    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7\" (UniqueName: \"kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7\") pod \"ef4f38ca-4642-4454-aa9d-24f7641b6874\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") "
Oct 30 22:44:02.360344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.360159    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca\") pod \"ef4f38ca-4642-4454-aa9d-24f7641b6874\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") "
Oct 30 22:44:02.360344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.360227    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca\") pod \"ef4f38ca-4642-4454-aa9d-24f7641b6874\" (UID: \"ef4f38ca-4642-4454-aa9d-24f7641b6874\") "
Oct 30 22:44:02.360897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.360867    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca" pod "ef4f38ca-4642-4454-aa9d-24f7641b6874" (UID: "ef4f38ca-4642-4454-aa9d-24f7641b6874"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:02.362953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.362921    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7" pod "ef4f38ca-4642-4454-aa9d-24f7641b6874" (UID: "ef4f38ca-4642-4454-aa9d-24f7641b6874"). InnerVolumeSpecName "kube-api-access-sctr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:02.362953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.362939    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca" pod "ef4f38ca-4642-4454-aa9d-24f7641b6874" (UID: "ef4f38ca-4642-4454-aa9d-24f7641b6874"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:02.461986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.461945    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctr7\" (UniqueName: \"kubernetes.io/projected/ef4f38ca-4642-4454-aa9d-24f7641b6874-kube-api-access-sctr7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:02.461986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.461987    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:02.462271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:02.462001    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4f38ca-4642-4454-aa9d-24f7641b6874-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.139622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.139584    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"
Oct 30 22:44:03.139622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.139610    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh" event={"ID":"ef4f38ca-4642-4454-aa9d-24f7641b6874","Type":"ContainerDied","Data":"41c45dc7cf1854237479a3805c454ddb9b68f7199e3262ae872e4b11df65e764"}
Oct 30 22:44:03.139936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.139656    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c45dc7cf1854237479a3805c454ddb9b68f7199e3262ae872e4b11df65e764"
Oct 30 22:44:03.490538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.490511    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:44:03.577579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577538    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.577812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577646    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.577812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577726    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.577943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577825    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.577943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577863    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.577943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577894    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.577943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577927    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:03.578166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.577952    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.578166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578015    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.578166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578094    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data\") pod \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\" (UID: \"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de\") "
Oct 30 22:44:03.578166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578117    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:03.578366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578304    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:03.578559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578522    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:03.578657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578640    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:03.578899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578878    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.578958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578906    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.578958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578923    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.578958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578937    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.578958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.578951    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.580211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.580185    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "kube-api-access-mdp4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:03.580304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.580191    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:44:03.580418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.580402    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:03.580465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.580450    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token" pod "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" (UID: "832c3337-d99c-4cbd-ba83-fbfa6d2ac0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:03.679944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.679912    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.679944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.679941    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.679944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.679952    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:03.680223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:03.679961    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdp4t\" (UniqueName: \"kubernetes.io/projected/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de-kube-api-access-mdp4t\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.151896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.151862    5028 generic.go:356] "Generic (PLEG): container finished" podID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerID="af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf" exitCode=0
Oct 30 22:44:04.152116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.151947    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"
Oct 30 22:44:04.152116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.151943    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" event={"ID":"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de","Type":"ContainerDied","Data":"af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf"}
Oct 30 22:44:04.152116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.152098    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8" event={"ID":"832c3337-d99c-4cbd-ba83-fbfa6d2ac0de","Type":"ContainerDied","Data":"abffdec49078abb84261cbc2e9c1e69ec1fd62be7cf4a364c04dc56587c737fa"}
Oct 30 22:44:04.152300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.152125    5028 scope.go:122] "RemoveContainer" containerID="af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf"
Oct 30 22:44:04.164061 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.164036    5028 scope.go:122] "RemoveContainer" containerID="89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"
Oct 30 22:44:04.174272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.174249    5028 scope.go:122] "RemoveContainer" containerID="af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf"
Oct 30 22:44:04.174537 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:44:04.174519    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf\": container with ID starting with af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf not found: ID does not exist" containerID="af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf"
Oct 30 22:44:04.174587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.174546    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf"} err="failed to get container status \"af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf\": rpc error: code = NotFound desc = could not find container \"af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf\": container with ID starting with af7627f193b33d8e58a5387799e0af4e066a6ef67b824f1893cf0eeedd90a7bf not found: ID does not exist"
Oct 30 22:44:04.174587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.174562    5028 scope.go:122] "RemoveContainer" containerID="89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"
Oct 30 22:44:04.174807 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:44:04.174792    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a\": container with ID starting with 89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a not found: ID does not exist" containerID="89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"
Oct 30 22:44:04.174851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.174810    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a"} err="failed to get container status \"89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a\": rpc error: code = NotFound desc = could not find container \"89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a\": container with ID starting with 89e912c66c2b7c1ea075725c3c3248d1226610581536285f05ab7d26a89b453a not found: ID does not exist"
Oct 30 22:44:04.362149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.362116    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"]
Oct 30 22:44:04.409038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:04.408966    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-771ff9-openshift-default-7c4689f4d9-6xfp8"]
Oct 30 22:44:05.145081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:05.145039    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:05.291638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:05.291600    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" path="/var/lib/kubelet/pods/832c3337-d99c-4cbd-ba83-fbfa6d2ac0de/volumes"
Oct 30 22:44:09.715437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:09.715388    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:11.408601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:11.408560    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"]
Oct 30 22:44:11.509737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:11.509701    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"]
Oct 30 22:44:11.767351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:11.767313    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:11.923952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:11.923905    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:14.239675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:14.239645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:14.612624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:14.612541    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:16.409678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.409628    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6" podUID="5dd292b8-d597-4f36-a8a0-fff7a36e94c9" containerName="payload-processing" containerID="cri-o://b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9" gracePeriod=25
Oct 30 22:44:16.709026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.709001    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:44:16.856741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.856710    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7\" (UniqueName: \"kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7\") pod \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") "
Oct 30 22:44:16.856937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.856824    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume\") pod \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\" (UID: \"5dd292b8-d597-4f36-a8a0-fff7a36e94c9\") "
Oct 30 22:44:16.857262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.857229    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume" pod "5dd292b8-d597-4f36-a8a0-fff7a36e94c9" (UID: "5dd292b8-d597-4f36-a8a0-fff7a36e94c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:16.857596 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.857566    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:16.863197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.863154    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7" pod "5dd292b8-d597-4f36-a8a0-fff7a36e94c9" (UID: "5dd292b8-d597-4f36-a8a0-fff7a36e94c9"). InnerVolumeSpecName "kube-api-access-7swh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:16.958790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:16.958757    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swh7\" (UniqueName: \"kubernetes.io/projected/5dd292b8-d597-4f36-a8a0-fff7a36e94c9-kube-api-access-7swh7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:17.218784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.218666    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8" podUID="f27c8960-61d7-4534-bede-92c649f1b644" containerName="payload-pre-processing" containerID="cri-o://5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea" gracePeriod=25
Oct 30 22:44:17.260510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.260476    5028 generic.go:356] "Generic (PLEG): container finished" podID="5dd292b8-d597-4f36-a8a0-fff7a36e94c9" containerID="b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9" exitCode=0
Oct 30 22:44:17.260678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.260534    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"
Oct 30 22:44:17.260678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.260550    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6" event={"ID":"5dd292b8-d597-4f36-a8a0-fff7a36e94c9","Type":"ContainerDied","Data":"b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9"}
Oct 30 22:44:17.260678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.260636    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6" event={"ID":"5dd292b8-d597-4f36-a8a0-fff7a36e94c9","Type":"ContainerDied","Data":"8e0f5b8ba7bf6ea1b98732acdf25cfee310586e0fe46fa2b0ec60124e42209f4"}
Oct 30 22:44:17.260678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.260662    5028 scope.go:122] "RemoveContainer" containerID="b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9"
Oct 30 22:44:17.339080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.339046    5028 scope.go:122] "RemoveContainer" containerID="b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9"
Oct 30 22:44:17.339823 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:44:17.339789    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9\": container with ID starting with b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9 not found: ID does not exist" containerID="b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9"
Oct 30 22:44:17.339957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.339835    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9"} err="failed to get container status \"b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9\": rpc error: code = NotFound desc = could not find container \"b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9\": container with ID starting with b48b3ae90e9f67cb6862664391134b33f605d3d202648e57f2ffa60d7f0dcac9 not found: ID does not exist"
Oct 30 22:44:17.423747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.423714    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"]
Oct 30 22:44:17.467400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.467366    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9fce61-7bfc755f9c-vjgk6"]
Oct 30 22:44:17.509175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.509151    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:44:17.565800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.565764    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm\" (UniqueName: \"kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm\") pod \"f27c8960-61d7-4534-bede-92c649f1b644\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") "
Oct 30 22:44:17.565996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.565812    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume\") pod \"f27c8960-61d7-4534-bede-92c649f1b644\" (UID: \"f27c8960-61d7-4534-bede-92c649f1b644\") "
Oct 30 22:44:17.566283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.566255    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume" pod "f27c8960-61d7-4534-bede-92c649f1b644" (UID: "f27c8960-61d7-4534-bede-92c649f1b644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:17.567955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.567933    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm" pod "f27c8960-61d7-4534-bede-92c649f1b644" (UID: "f27c8960-61d7-4534-bede-92c649f1b644"). InnerVolumeSpecName "kube-api-access-prvxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:17.667604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.667565    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvxm\" (UniqueName: \"kubernetes.io/projected/f27c8960-61d7-4534-bede-92c649f1b644-kube-api-access-prvxm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:17.667604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:17.667603    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c8960-61d7-4534-bede-92c649f1b644-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:18.269600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.269562    5028 generic.go:356] "Generic (PLEG): container finished" podID="f27c8960-61d7-4534-bede-92c649f1b644" containerID="5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea" exitCode=0
Oct 30 22:44:18.269820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.269622    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"
Oct 30 22:44:18.269820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.269651    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8" event={"ID":"f27c8960-61d7-4534-bede-92c649f1b644","Type":"ContainerDied","Data":"5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea"}
Oct 30 22:44:18.269820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.269734    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8" event={"ID":"f27c8960-61d7-4534-bede-92c649f1b644","Type":"ContainerDied","Data":"868c1655ed326f71bf3a3679bc7e3d75bb1a6c5e1f12359def153dd4c9a9c822"}
Oct 30 22:44:18.269820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.269768    5028 scope.go:122] "RemoveContainer" containerID="5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea"
Oct 30 22:44:18.281203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.281182    5028 scope.go:122] "RemoveContainer" containerID="5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea"
Oct 30 22:44:18.281467 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:44:18.281446    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea\": container with ID starting with 5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea not found: ID does not exist" containerID="5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea"
Oct 30 22:44:18.281524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.281475    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea"} err="failed to get container status \"5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea\": rpc error: code = NotFound desc = could not find container \"5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea\": container with ID starting with 5440a59dead64f10a22dcd19f984f3161a82ecd4348ab7225504546e472e9fea not found: ID does not exist"
Oct 30 22:44:18.432227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.432177    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"]
Oct 30 22:44:18.438083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.438048    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9fce61-86b586b655-x4pz8"]
Oct 30 22:44:18.687334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.687243    5028 scope.go:122] "RemoveContainer" containerID="16c458cf99ee5b39e46fb293b2d14ebf3f3965bec7ac97e90342bad54cd8b3ef"
Oct 30 22:44:18.697804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.697783    5028 scope.go:122] "RemoveContainer" containerID="ebaae30b26137607f4459fc873068151c624e13f68456766a978712c28726f61"
Oct 30 22:44:18.738953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:18.738827    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:19.049148 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.049109    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"]
Oct 30 22:44:19.055387 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.055345    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"]
Oct 30 22:44:19.060978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.060953    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648961eb-sb9j8"]
Oct 30 22:44:19.095876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.095846    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fcc509-2d47p"]
Oct 30 22:44:19.292020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.291984    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b67eee3-3b75-4237-b2f7-0261fff98e2d" path="/var/lib/kubelet/pods/2b67eee3-3b75-4237-b2f7-0261fff98e2d/volumes"
Oct 30 22:44:19.292388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.292373    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd292b8-d597-4f36-a8a0-fff7a36e94c9" path="/var/lib/kubelet/pods/5dd292b8-d597-4f36-a8a0-fff7a36e94c9/volumes"
Oct 30 22:44:19.292819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.292804    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbbb719-03b2-4c1c-828f-4373fe8996c7" path="/var/lib/kubelet/pods/8dbbb719-03b2-4c1c-828f-4373fe8996c7/volumes"
Oct 30 22:44:19.293465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:19.293450    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27c8960-61d7-4534-bede-92c649f1b644" path="/var/lib/kubelet/pods/f27c8960-61d7-4534-bede-92c649f1b644/volumes"
Oct 30 22:44:23.236933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:23.236895    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:27.743156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:27.743118    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:28.441435 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:44:28.441394    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:44:32.239010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:32.238963    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:35.419001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:35.418971    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:35.579822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:35.579781    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:36.759270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:36.759239    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:38.067208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:38.067174    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:41.238621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:41.238581    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:45.776987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:45.776945    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:47.050884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:47.050845    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"]
Oct 30 22:44:47.097696 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:47.097655    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d7d79d-8zlj9"]
Oct 30 22:44:47.293160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:47.293129    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac38bc2-6733-4594-831a-e07764f2e3ce" path="/var/lib/kubelet/pods/8ac38bc2-6733-4594-831a-e07764f2e3ce/volumes"
Oct 30 22:44:48.049770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:48.049730    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"]
Oct 30 22:44:48.096391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:48.096356    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240bacd2-bqhs4"]
Oct 30 22:44:49.292906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:49.292861    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dfad38-3130-45a1-8ad7-5147d62da656" path="/var/lib/kubelet/pods/57dfad38-3130-45a1-8ad7-5147d62da656/volumes"
Oct 30 22:44:50.325541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:50.325505    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:54.839077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:54.839046    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:58.435963 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:44:58.435918    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:44:59.348763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:44:59.348733    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:00.176370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.176334    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"]
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179760    5028 cpu_manager.go:414] "containerMap: removing container" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179800    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179843    5028 cpu_manager.go:414] "containerMap: removing container" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179854    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179903    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f27c8960-61d7-4534-bede-92c649f1b644" containerName="payload-pre-processing"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179913    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27c8960-61d7-4534-bede-92c649f1b644" containerName="payload-pre-processing"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179968    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ef4f38ca-4642-4454-aa9d-24f7641b6874" containerName="revoke-keys"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.179979    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4f38ca-4642-4454-aa9d-24f7641b6874" containerName="revoke-keys"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180001    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5dd292b8-d597-4f36-a8a0-fff7a36e94c9" containerName="payload-processing"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180017    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd292b8-d597-4f36-a8a0-fff7a36e94c9" containerName="payload-processing"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180633    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180664    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="832c3337-d99c-4cbd-ba83-fbfa6d2ac0de" containerName="istio-proxy"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180680    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27c8960-61d7-4534-bede-92c649f1b644" containerName="payload-pre-processing"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180721    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4f38ca-4642-4454-aa9d-24f7641b6874" containerName="revoke-keys"
Oct 30 22:45:00.186081 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.180732    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd292b8-d597-4f36-a8a0-fff7a36e94c9" containerName="payload-processing"
Oct 30 22:45:00.192905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.192870    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"]
Oct 30 22:45:00.193107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.193082    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.198184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.198154    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"]
Oct 30 22:45:00.198342 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.198281    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.202170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.202151    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.224548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.224518    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"]
Oct 30 22:45:00.228678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.228652    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"]
Oct 30 22:45:00.228877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.228860    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.232933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.232910    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"]
Oct 30 22:45:00.233205 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.233186    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.237855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.237832    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"]
Oct 30 22:45:00.238001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.237950    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.242706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.242670    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.243756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.243735    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.244458 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.244411    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.269758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.269723    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tl2\" (UniqueName: \"kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2\") pod \"maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.269928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.269912    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.270027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.269979    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfsx\" (UniqueName: \"kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx\") pod \"maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.270076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.270051    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgm6v\" (UniqueName: \"kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v\") pod \"maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.270175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.270154    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.270300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.270283    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.272213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.272189    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"]
Oct 30 22:45:00.275833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.275816    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"]
Oct 30 22:45:00.279222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.279203    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"]
Oct 30 22:45:00.282278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.282258    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"]
Oct 30 22:45:00.320965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.320927    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"]
Oct 30 22:45:00.333268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.333234    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"]
Oct 30 22:45:00.337593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.337565    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:45:00.338579 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.338553    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"]
Oct 30 22:45:00.371278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371219    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkpn\" (UniqueName: \"kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.371496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371348    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.371496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371406    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tl2\" (UniqueName: \"kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2\") pod \"maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.371496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371467    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.371717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371577    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692vb\" (UniqueName: \"kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.371717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371629    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.371717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371668    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqs9k\" (UniqueName: \"kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k\") pod \"maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.371907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371786    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.371977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371923    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfsx\" (UniqueName: \"kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx\") pod \"maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.371977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371968    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzf5\" (UniqueName: \"kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.372110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.371996    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgm6v\" (UniqueName: \"kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v\") pod \"maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.372110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.372075    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.372230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.372111    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.372230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.372217    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.372583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.372556    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.372886 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.372854    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.372973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.372912    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.430734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.430626    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tl2\" (UniqueName: \"kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2\") pod \"maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.430947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.430926    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgm6v\" (UniqueName: \"kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v\") pod \"maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.431034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.431015    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfsx\" (UniqueName: \"kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx\") pod \"maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.474174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474141    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzf5\" (UniqueName: \"kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.474337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474187    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.474394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474370    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.474431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474414    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkpn\" (UniqueName: \"kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.474545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474529    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.474591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474581    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692vb\" (UniqueName: \"kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.474651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474617    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.474745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.474655    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqs9k\" (UniqueName: \"kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k\") pod \"maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.475085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.475062    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.475223 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.475196    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.475319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.475300    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.475377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.475363    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.510562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.510526    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:00.521059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.521035    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:00.528780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.528724    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:00.542747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.542368    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzf5\" (UniqueName: \"kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5\") pod \"maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.542747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.542674    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkpn\" (UniqueName: \"kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.543101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.543072    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692vb\" (UniqueName: \"kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.543753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.543708    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqs9k\" (UniqueName: \"kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k\") pod \"maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.553094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.553066    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:00.560747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.560089    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:00.570147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.570119    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:00.593022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.583149    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:00.908146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.908104    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"]
Oct 30 22:45:00.909300 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:00.909269    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea0264c_eae4_485c_8d85_bc902d20d7f6.slice/crio-05434d23497e8e4286ed3509ddb5b36db584210fff27f7f97d5e00c83e7f7931 WatchSource:0}: Error finding container 05434d23497e8e4286ed3509ddb5b36db584210fff27f7f97d5e00c83e7f7931: Status 404 returned error can't find the container with id 05434d23497e8e4286ed3509ddb5b36db584210fff27f7f97d5e00c83e7f7931
Oct 30 22:45:00.944903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:00.944871    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"]
Oct 30 22:45:00.949145 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:00.949120    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cd58ff_0e7d_4146_8357_d93b8c5dc5cc.slice/crio-c5bfbcb8b379a6c6018950ae7b40369b1c1815ca79ab4121beb00508c485f17d WatchSource:0}: Error finding container c5bfbcb8b379a6c6018950ae7b40369b1c1815ca79ab4121beb00508c485f17d: Status 404 returned error can't find the container with id c5bfbcb8b379a6c6018950ae7b40369b1c1815ca79ab4121beb00508c485f17d
Oct 30 22:45:01.034520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.034493    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"]
Oct 30 22:45:01.038903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.038880    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"]
Oct 30 22:45:01.076444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.076415    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"]
Oct 30 22:45:01.120264 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:01.120217    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81944d12_4147_4747_bd07_baaa00491a29.slice/crio-83ca7422673da2599899165338425780f97b4006623bf62bda4b498a5ccdf3db WatchSource:0}: Error finding container 83ca7422673da2599899165338425780f97b4006623bf62bda4b498a5ccdf3db: Status 404 returned error can't find the container with id 83ca7422673da2599899165338425780f97b4006623bf62bda4b498a5ccdf3db
Oct 30 22:45:01.121395 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:01.121359    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacac90f_1c25_4101_a6d3_4fb23a9d406a.slice/crio-9a7e144aec361b2af88e8e174b4b78dcb260b661aaee5a78c6e55b772d475c87 WatchSource:0}: Error finding container 9a7e144aec361b2af88e8e174b4b78dcb260b661aaee5a78c6e55b772d475c87: Status 404 returned error can't find the container with id 9a7e144aec361b2af88e8e174b4b78dcb260b661aaee5a78c6e55b772d475c87
Oct 30 22:45:01.122429 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:01.122393    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab923bcb_05ec_462e_a313_9b4308a15d3f.slice/crio-71e33e68ed512dc9bea886b2c053fbea329f1abb7659d6da7944de734ac983b2 WatchSource:0}: Error finding container 71e33e68ed512dc9bea886b2c053fbea329f1abb7659d6da7944de734ac983b2: Status 404 returned error can't find the container with id 71e33e68ed512dc9bea886b2c053fbea329f1abb7659d6da7944de734ac983b2
Oct 30 22:45:01.147049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.147021    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"]
Oct 30 22:45:01.153005 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:01.152949    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c418b0_98ab_48f4_858d_a80d0da2da61.slice/crio-f10ad5b30c471d845c6de0c01d20f4fdf3ba30bcddd654f84d84b9be6aae32be WatchSource:0}: Error finding container f10ad5b30c471d845c6de0c01d20f4fdf3ba30bcddd654f84d84b9be6aae32be: Status 404 returned error can't find the container with id f10ad5b30c471d845c6de0c01d20f4fdf3ba30bcddd654f84d84b9be6aae32be
Oct 30 22:45:01.162386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.161046    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"]
Oct 30 22:45:01.638981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.638823    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerStarted","Data":"9a7e144aec361b2af88e8e174b4b78dcb260b661aaee5a78c6e55b772d475c87"}
Oct 30 22:45:01.640998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.640967    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerStarted","Data":"71e33e68ed512dc9bea886b2c053fbea329f1abb7659d6da7944de734ac983b2"}
Oct 30 22:45:01.642743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.642713    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerStarted","Data":"71ffd0b7b3ed1ee7c6b519d024cfc845180c356e81749d45f809681aaf6bc3bd"}
Oct 30 22:45:01.642853 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.642767    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerStarted","Data":"05434d23497e8e4286ed3509ddb5b36db584210fff27f7f97d5e00c83e7f7931"}
Oct 30 22:45:01.644219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.644197    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerStarted","Data":"9f1a49e9194cccd28869f5a37f2775e0306e5fbc79a2ddce32e35dd5dd9f670a"}
Oct 30 22:45:01.644321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.644233    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerStarted","Data":"c5bfbcb8b379a6c6018950ae7b40369b1c1815ca79ab4121beb00508c485f17d"}
Oct 30 22:45:01.645746 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.645727    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerStarted","Data":"5ac0e9d69fe799777871f210d3c86816af96e834666b7ea9c6643586fa9bea18"}
Oct 30 22:45:01.647285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.647264    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerStarted","Data":"83ca7422673da2599899165338425780f97b4006623bf62bda4b498a5ccdf3db"}
Oct 30 22:45:01.648675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.648655    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerStarted","Data":"f10ad5b30c471d845c6de0c01d20f4fdf3ba30bcddd654f84d84b9be6aae32be"}
Oct 30 22:45:01.756413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.756358    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" podStartSLOduration=1.7563447349999999 podStartE2EDuration="1.756344735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.75305418 +0000 UTC m=+2632.602567174" watchObservedRunningTime="2026-10-30 22:45:01.756344735 +0000 UTC m=+2632.605857686"
Oct 30 22:45:01.866580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.866521    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" podStartSLOduration=1.866507747 podStartE2EDuration="1.866507747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.862865926 +0000 UTC m=+2632.712378899" watchObservedRunningTime="2026-10-30 22:45:01.866507747 +0000 UTC m=+2632.716020720"
Oct 30 22:45:01.975704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:01.975636    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" podStartSLOduration=1.975621477 podStartE2EDuration="1.975621477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.974956493 +0000 UTC m=+2632.824469466" watchObservedRunningTime="2026-10-30 22:45:01.975621477 +0000 UTC m=+2632.825134444"
Oct 30 22:45:02.087925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.087823    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" podStartSLOduration=2.087808386 podStartE2EDuration="2.087808386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.08639979 +0000 UTC m=+2632.935912786" watchObservedRunningTime="2026-10-30 22:45:02.087808386 +0000 UTC m=+2632.937321358"
Oct 30 22:45:02.657936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.657844    5028 generic.go:356] "Generic (PLEG): container finished" podID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerID="71ffd0b7b3ed1ee7c6b519d024cfc845180c356e81749d45f809681aaf6bc3bd" exitCode=60
Oct 30 22:45:02.658405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.657935    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerDied","Data":"71ffd0b7b3ed1ee7c6b519d024cfc845180c356e81749d45f809681aaf6bc3bd"}
Oct 30 22:45:02.658405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.658212    5028 scope.go:122] "RemoveContainer" containerID="71ffd0b7b3ed1ee7c6b519d024cfc845180c356e81749d45f809681aaf6bc3bd"
Oct 30 22:45:02.659888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.659860    5028 generic.go:356] "Generic (PLEG): container finished" podID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerID="9f1a49e9194cccd28869f5a37f2775e0306e5fbc79a2ddce32e35dd5dd9f670a" exitCode=60
Oct 30 22:45:02.660096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.659952    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerDied","Data":"9f1a49e9194cccd28869f5a37f2775e0306e5fbc79a2ddce32e35dd5dd9f670a"}
Oct 30 22:45:02.660281 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.660206    5028 scope.go:122] "RemoveContainer" containerID="9f1a49e9194cccd28869f5a37f2775e0306e5fbc79a2ddce32e35dd5dd9f670a"
Oct 30 22:45:02.661673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.661652    5028 generic.go:356] "Generic (PLEG): container finished" podID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerID="41d45af0e069e8cc4f9dee1930e8111c64738005a575d2c4a1d3294c84324c60" exitCode=60
Oct 30 22:45:02.661829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.661811    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerDied","Data":"41d45af0e069e8cc4f9dee1930e8111c64738005a575d2c4a1d3294c84324c60"}
Oct 30 22:45:02.662105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.662000    5028 scope.go:122] "RemoveContainer" containerID="41d45af0e069e8cc4f9dee1930e8111c64738005a575d2c4a1d3294c84324c60"
Oct 30 22:45:02.663678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.663659    5028 generic.go:356] "Generic (PLEG): container finished" podID="81944d12-4147-4747-bd07-baaa00491a29" containerID="6afad8dd899fc8b898c1cda50359f2801246ab3e0fcf77a9255bf549e4c5a9b9" exitCode=60
Oct 30 22:45:02.663767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.663715    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerDied","Data":"6afad8dd899fc8b898c1cda50359f2801246ab3e0fcf77a9255bf549e4c5a9b9"}
Oct 30 22:45:02.663949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.663932    5028 scope.go:122] "RemoveContainer" containerID="6afad8dd899fc8b898c1cda50359f2801246ab3e0fcf77a9255bf549e4c5a9b9"
Oct 30 22:45:02.665584 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.665563    5028 generic.go:356] "Generic (PLEG): container finished" podID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerID="40706a5e1dcffea16fd6523ba29a4c208a85b1e01153db48c0b2303fc1fa9208" exitCode=60
Oct 30 22:45:02.665739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.665657    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerDied","Data":"40706a5e1dcffea16fd6523ba29a4c208a85b1e01153db48c0b2303fc1fa9208"}
Oct 30 22:45:02.665914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.665896    5028 scope.go:122] "RemoveContainer" containerID="40706a5e1dcffea16fd6523ba29a4c208a85b1e01153db48c0b2303fc1fa9208"
Oct 30 22:45:02.667542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.667521    5028 generic.go:356] "Generic (PLEG): container finished" podID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerID="cdb0229558df0b49ee4bca74ab3e602371041f64ac409c5e9fcec4dc936fb712" exitCode=60
Oct 30 22:45:02.667902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.667604    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerDied","Data":"cdb0229558df0b49ee4bca74ab3e602371041f64ac409c5e9fcec4dc936fb712"}
Oct 30 22:45:02.667902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.667826    5028 scope.go:122] "RemoveContainer" containerID="cdb0229558df0b49ee4bca74ab3e602371041f64ac409c5e9fcec4dc936fb712"
Oct 30 22:45:02.669413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.669390    5028 generic.go:356] "Generic (PLEG): container finished" podID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerID="4292b20ec9ddd374fbc6c9b2cb79c8f79bc6e42239b6c1b8a6c231261afc6d36" exitCode=60
Oct 30 22:45:02.669508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.669481    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerDied","Data":"4292b20ec9ddd374fbc6c9b2cb79c8f79bc6e42239b6c1b8a6c231261afc6d36"}
Oct 30 22:45:02.669715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:02.669671    5028 scope.go:122] "RemoveContainer" containerID="4292b20ec9ddd374fbc6c9b2cb79c8f79bc6e42239b6c1b8a6c231261afc6d36"
Oct 30 22:45:03.680257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.680150    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerStarted","Data":"777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726"}
Oct 30 22:45:03.682216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.682169    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerStarted","Data":"1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529"}
Oct 30 22:45:03.684308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.684284    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerStarted","Data":"bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948"}
Oct 30 22:45:03.686338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.686312    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerStarted","Data":"7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840"}
Oct 30 22:45:03.688362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.688341    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerStarted","Data":"a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218"}
Oct 30 22:45:03.690434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.690408    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerStarted","Data":"09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32"}
Oct 30 22:45:03.692523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.692501    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerStarted","Data":"43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7"}
Oct 30 22:45:03.944878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:03.944844    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:04.129310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.129249    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" podStartSLOduration=4.129234228 podStartE2EDuration="4.129234228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.125324705 +0000 UTC m=+2634.974837681" watchObservedRunningTime="2026-10-30 22:45:04.129234228 +0000 UTC m=+2634.978747200"
Oct 30 22:45:04.238225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.238164    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" podStartSLOduration=4.238145808 podStartE2EDuration="4.238145808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.237986916 +0000 UTC m=+2635.087499889" watchObservedRunningTime="2026-10-30 22:45:04.238145808 +0000 UTC m=+2635.087658787"
Oct 30 22:45:04.353907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.353848    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" podStartSLOduration=4.353832701 podStartE2EDuration="4.353832701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.350210403 +0000 UTC m=+2635.199723377" watchObservedRunningTime="2026-10-30 22:45:04.353832701 +0000 UTC m=+2635.203345674"
Oct 30 22:45:04.705383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.705296    5028 generic.go:356] "Generic (PLEG): container finished" podID="81944d12-4147-4747-bd07-baaa00491a29" containerID="43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7" exitCode=60
Oct 30 22:45:04.705860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.705385    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerDied","Data":"43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7"}
Oct 30 22:45:04.705860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.705463    5028 scope.go:122] "RemoveContainer" containerID="6afad8dd899fc8b898c1cda50359f2801246ab3e0fcf77a9255bf549e4c5a9b9"
Oct 30 22:45:04.705948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.705873    5028 scope.go:122] "RemoveContainer" containerID="43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7"
Oct 30 22:45:04.706277 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.706248    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld_odh-ai-gateway-infra(81944d12-4147-4747-bd07-baaa00491a29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" podUID="81944d12-4147-4747-bd07-baaa00491a29"
Oct 30 22:45:04.708487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.708461    5028 generic.go:356] "Generic (PLEG): container finished" podID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerID="777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726" exitCode=60
Oct 30 22:45:04.708607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.708545    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerDied","Data":"777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726"}
Oct 30 22:45:04.708889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.708868    5028 scope.go:122] "RemoveContainer" containerID="777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726"
Oct 30 22:45:04.709232 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.709204    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl_odh-ai-gateway-infra(00c418b0-98ab-48f4-858d-a80d0da2da61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61"
Oct 30 22:45:04.716520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.716489    5028 generic.go:356] "Generic (PLEG): container finished" podID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerID="1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529" exitCode=60
Oct 30 22:45:04.716667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.716613    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerDied","Data":"1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529"}
Oct 30 22:45:04.717919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.717058    5028 scope.go:122] "RemoveContainer" containerID="1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529"
Oct 30 22:45:04.717919 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.717421    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh_odh-ai-gateway-infra(cacac90f-1c25-4101-a6d3-4fb23a9d406a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a"
Oct 30 22:45:04.719657 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.719634    5028 generic.go:356] "Generic (PLEG): container finished" podID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerID="bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948" exitCode=60
Oct 30 22:45:04.719830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.719810    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerDied","Data":"bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948"}
Oct 30 22:45:04.720235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.720216    5028 scope.go:122] "RemoveContainer" containerID="bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948"
Oct 30 22:45:04.720568 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.720541    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r_odh-ai-gateway-infra(ab923bcb-05ec-462e-a313-9b4308a15d3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f"
Oct 30 22:45:04.726113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.726082    5028 generic.go:356] "Generic (PLEG): container finished" podID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerID="7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840" exitCode=60
Oct 30 22:45:04.726850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.726303    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerDied","Data":"7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840"}
Oct 30 22:45:04.727003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.726662    5028 scope.go:122] "RemoveContainer" containerID="7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840"
Oct 30 22:45:04.727445 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.727417    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9_odh-ai-gateway-infra(aea0264c-eae4-485c-8d85-bc902d20d7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6"
Oct 30 22:45:04.732335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.732315    5028 generic.go:356] "Generic (PLEG): container finished" podID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerID="a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218" exitCode=60
Oct 30 22:45:04.732437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.732394    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerDied","Data":"a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218"}
Oct 30 22:45:04.732819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.732759    5028 scope.go:122] "RemoveContainer" containerID="a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218"
Oct 30 22:45:04.732890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.732840    5028 scope.go:122] "RemoveContainer" containerID="40706a5e1dcffea16fd6523ba29a4c208a85b1e01153db48c0b2303fc1fa9208"
Oct 30 22:45:04.733121 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.733093    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb_odh-ai-gateway-infra(43cd58ff-0e7d-4146-8357-d93b8c5dc5cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc"
Oct 30 22:45:04.735037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.735015    5028 generic.go:356] "Generic (PLEG): container finished" podID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerID="09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32" exitCode=60
Oct 30 22:45:04.735127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.735065    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerDied","Data":"09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32"}
Oct 30 22:45:04.735363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.735345    5028 scope.go:122] "RemoveContainer" containerID="09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32"
Oct 30 22:45:04.735675 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:04.735654    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp_odh-ai-gateway-infra(6d3791f8-4e73-4524-a52f-36ad674307b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5"
Oct 30 22:45:04.749808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.749779    5028 scope.go:122] "RemoveContainer" containerID="cdb0229558df0b49ee4bca74ab3e602371041f64ac409c5e9fcec4dc936fb712"
Oct 30 22:45:04.765642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.765616    5028 scope.go:122] "RemoveContainer" containerID="4292b20ec9ddd374fbc6c9b2cb79c8f79bc6e42239b6c1b8a6c231261afc6d36"
Oct 30 22:45:04.779645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.779619    5028 scope.go:122] "RemoveContainer" containerID="71ffd0b7b3ed1ee7c6b519d024cfc845180c356e81749d45f809681aaf6bc3bd"
Oct 30 22:45:04.793261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.793238    5028 scope.go:122] "RemoveContainer" containerID="9f1a49e9194cccd28869f5a37f2775e0306e5fbc79a2ddce32e35dd5dd9f670a"
Oct 30 22:45:04.806531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:04.806511    5028 scope.go:122] "RemoveContainer" containerID="41d45af0e069e8cc4f9dee1930e8111c64738005a575d2c4a1d3294c84324c60"
Oct 30 22:45:05.746709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.746491    5028 scope.go:122] "RemoveContainer" containerID="09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32"
Oct 30 22:45:05.747206 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.746847    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp_odh-ai-gateway-infra(6d3791f8-4e73-4524-a52f-36ad674307b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5"
Oct 30 22:45:05.748448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.748421    5028 scope.go:122] "RemoveContainer" containerID="43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7"
Oct 30 22:45:05.748776 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.748747    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld_odh-ai-gateway-infra(81944d12-4147-4747-bd07-baaa00491a29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" podUID="81944d12-4147-4747-bd07-baaa00491a29"
Oct 30 22:45:05.750296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.750276    5028 scope.go:122] "RemoveContainer" containerID="777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726"
Oct 30 22:45:05.750597 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.750573    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl_odh-ai-gateway-infra(00c418b0-98ab-48f4-858d-a80d0da2da61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61"
Oct 30 22:45:05.752124 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.752103    5028 scope.go:122] "RemoveContainer" containerID="1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529"
Oct 30 22:45:05.752407 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.752384    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh_odh-ai-gateway-infra(cacac90f-1c25-4101-a6d3-4fb23a9d406a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a"
Oct 30 22:45:05.753840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.753822    5028 scope.go:122] "RemoveContainer" containerID="bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948"
Oct 30 22:45:05.754113 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.754090    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r_odh-ai-gateway-infra(ab923bcb-05ec-462e-a313-9b4308a15d3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f"
Oct 30 22:45:05.755626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.755607    5028 scope.go:122] "RemoveContainer" containerID="7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840"
Oct 30 22:45:05.755904 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.755882    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9_odh-ai-gateway-infra(aea0264c-eae4-485c-8d85-bc902d20d7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6"
Oct 30 22:45:05.757532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:05.757514    5028 scope.go:122] "RemoveContainer" containerID="a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218"
Oct 30 22:45:05.757789 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:05.757767    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb_odh-ai-gateway-infra(43cd58ff-0e7d-4146-8357-d93b8c5dc5cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc"
Oct 30 22:45:08.445448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:08.445413    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:12.050929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:12.050892    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"]
Oct 30 22:45:12.098414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:12.098376    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"]
Oct 30 22:45:12.101953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:12.101929    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9afb2d3c-p84d7"]
Oct 30 22:45:12.105382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:12.105352    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c099526-7n6kd"]
Oct 30 22:45:12.947405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:12.947354    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:13.292465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:13.292423    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058f959b-ad04-45b9-9162-f3392a85a7ad" path="/var/lib/kubelet/pods/058f959b-ad04-45b9-9162-f3392a85a7ad/volumes"
Oct 30 22:45:13.293106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:13.293086    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87db2e32-aa79-409b-b1f9-2a9a50dc347e" path="/var/lib/kubelet/pods/87db2e32-aa79-409b-b1f9-2a9a50dc347e/volumes"
Oct 30 22:45:14.283391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:14.283358    5028 scope.go:122] "RemoveContainer" containerID="09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32"
Oct 30 22:45:14.840964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:14.840870    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerStarted","Data":"320a8d47b6c91dbf8e38ae9637867677d76cb3c983a8575836c2dd5e0f8f9710"}
Oct 30 22:45:14.961921 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:14.961874    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:15.283888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.283856    5028 scope.go:122] "RemoveContainer" containerID="bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948"
Oct 30 22:45:15.284085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.283975    5028 scope.go:122] "RemoveContainer" containerID="a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218"
Oct 30 22:45:15.284085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.284064    5028 scope.go:122] "RemoveContainer" containerID="43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7"
Oct 30 22:45:15.851996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.851910    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerStarted","Data":"e97348130bd0ef59a188f4f22426c0d46adb7dc0b51f80d13d7de19254fdff67"}
Oct 30 22:45:15.854230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.854195    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerStarted","Data":"e68af4908900b3a5a1caa636ad53443d1ebebb127c0243f91cc45ecfbae4e86f"}
Oct 30 22:45:15.856117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.856091    5028 generic.go:356] "Generic (PLEG): container finished" podID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerID="320a8d47b6c91dbf8e38ae9637867677d76cb3c983a8575836c2dd5e0f8f9710" exitCode=60
Oct 30 22:45:15.856245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.856159    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerDied","Data":"320a8d47b6c91dbf8e38ae9637867677d76cb3c983a8575836c2dd5e0f8f9710"}
Oct 30 22:45:15.856245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.856215    5028 scope.go:122] "RemoveContainer" containerID="09ce211e8e955b37a88d520c740b5ccdd001d364f9c0ba93d210217f5d054d32"
Oct 30 22:45:15.856575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.856554    5028 scope.go:122] "RemoveContainer" containerID="320a8d47b6c91dbf8e38ae9637867677d76cb3c983a8575836c2dd5e0f8f9710"
Oct 30 22:45:15.856939 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:15.856916    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp_odh-ai-gateway-infra(6d3791f8-4e73-4524-a52f-36ad674307b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5"
Oct 30 22:45:15.858455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.858429    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerStarted","Data":"7d749c86eab6f436af450bcb9ea416cd45d5167c7f411b516cc40627875c076a"}
Oct 30 22:45:15.966363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:15.966328    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"]
Oct 30 22:45:16.871000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.870968    5028 generic.go:356] "Generic (PLEG): container finished" podID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerID="e97348130bd0ef59a188f4f22426c0d46adb7dc0b51f80d13d7de19254fdff67" exitCode=60
Oct 30 22:45:16.871446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.871043    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerDied","Data":"e97348130bd0ef59a188f4f22426c0d46adb7dc0b51f80d13d7de19254fdff67"}
Oct 30 22:45:16.871446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.871122    5028 scope.go:122] "RemoveContainer" containerID="bc80b1d28e623f8258d085d5272ba8d055b2cc437978c9748699ea19fbf36948"
Oct 30 22:45:16.871610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.871506    5028 scope.go:122] "RemoveContainer" containerID="e97348130bd0ef59a188f4f22426c0d46adb7dc0b51f80d13d7de19254fdff67"
Oct 30 22:45:16.871874 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:16.871843    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r_odh-ai-gateway-infra(ab923bcb-05ec-462e-a313-9b4308a15d3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f"
Oct 30 22:45:16.873552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.873531    5028 generic.go:356] "Generic (PLEG): container finished" podID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerID="e68af4908900b3a5a1caa636ad53443d1ebebb127c0243f91cc45ecfbae4e86f" exitCode=60
Oct 30 22:45:16.873655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.873609    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerDied","Data":"e68af4908900b3a5a1caa636ad53443d1ebebb127c0243f91cc45ecfbae4e86f"}
Oct 30 22:45:16.873979 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.873961    5028 scope.go:122] "RemoveContainer" containerID="e68af4908900b3a5a1caa636ad53443d1ebebb127c0243f91cc45ecfbae4e86f"
Oct 30 22:45:16.874287 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:16.874261    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb_odh-ai-gateway-infra(43cd58ff-0e7d-4146-8357-d93b8c5dc5cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc"
Oct 30 22:45:16.877531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.877510    5028 generic.go:356] "Generic (PLEG): container finished" podID="81944d12-4147-4747-bd07-baaa00491a29" containerID="7d749c86eab6f436af450bcb9ea416cd45d5167c7f411b516cc40627875c076a" exitCode=60
Oct 30 22:45:16.877641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.877578    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerDied","Data":"7d749c86eab6f436af450bcb9ea416cd45d5167c7f411b516cc40627875c076a"}
Oct 30 22:45:16.877898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.877882    5028 scope.go:122] "RemoveContainer" containerID="7d749c86eab6f436af450bcb9ea416cd45d5167c7f411b516cc40627875c076a"
Oct 30 22:45:16.878137 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:16.878121    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld_odh-ai-gateway-infra(81944d12-4147-4747-bd07-baaa00491a29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" podUID="81944d12-4147-4747-bd07-baaa00491a29"
Oct 30 22:45:16.886840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.886819    5028 scope.go:122] "RemoveContainer" containerID="a25d093481afe985baaf62eeec323c93a3817eae672035df23aefd92de435218"
Oct 30 22:45:16.901486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.901448    5028 scope.go:122] "RemoveContainer" containerID="43db3225c8b255535e902d6a44be57785ff8a650f09c4177b02d9394374349e7"
Oct 30 22:45:16.977501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:16.977459    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"]
Oct 30 22:45:17.080030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.079991    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:17.088486 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.088454    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"]
Oct 30 22:45:17.198477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.198381    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca\") pod \"6d3791f8-4e73-4524-a52f-36ad674307b5\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") "
Oct 30 22:45:17.198671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.198618    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5\" (UniqueName: \"kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5\") pod \"6d3791f8-4e73-4524-a52f-36ad674307b5\" (UID: \"6d3791f8-4e73-4524-a52f-36ad674307b5\") "
Oct 30 22:45:17.198793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.198772    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca" pod "6d3791f8-4e73-4524-a52f-36ad674307b5" (UID: "6d3791f8-4e73-4524-a52f-36ad674307b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.199103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.199081    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3791f8-4e73-4524-a52f-36ad674307b5-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.200914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.200893    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5" pod "6d3791f8-4e73-4524-a52f-36ad674307b5" (UID: "6d3791f8-4e73-4524-a52f-36ad674307b5"). InnerVolumeSpecName "kube-api-access-stzf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.284284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.284255    5028 scope.go:122] "RemoveContainer" containerID="777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726"
Oct 30 22:45:17.284549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.284516    5028 scope.go:122] "RemoveContainer" containerID="7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840"
Oct 30 22:45:17.284729 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.284581    5028 scope.go:122] "RemoveContainer" containerID="1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529"
Oct 30 22:45:17.299902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.299878    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stzf5\" (UniqueName: \"kubernetes.io/projected/6d3791f8-4e73-4524-a52f-36ad674307b5-kube-api-access-stzf5\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.310251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.310229    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"]
Oct 30 22:45:17.450922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.450827    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:17.888897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.888873    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"
Oct 30 22:45:17.889307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.888904    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp" event={"ID":"6d3791f8-4e73-4524-a52f-36ad674307b5","Type":"ContainerDied","Data":"5ac0e9d69fe799777871f210d3c86816af96e834666b7ea9c6643586fa9bea18"}
Oct 30 22:45:17.889307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.889007    5028 scope.go:122] "RemoveContainer" containerID="320a8d47b6c91dbf8e38ae9637867677d76cb3c983a8575836c2dd5e0f8f9710"
Oct 30 22:45:17.894022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.893996    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerStarted","Data":"894999c7400d56a2ae796eb866e1638315caceabda469cc09f0b9db615e5ea88"}
Oct 30 22:45:17.896376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.896353    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerStarted","Data":"158a0f46e9b054d2f9a8b48490f0c088e103d0d06d096696368263195da60909"}
Oct 30 22:45:17.907779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:17.907746    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerStarted","Data":"a2cdfd056622941721a615db338e1dd31ed10edea13a081c9adc797dc31456f9"}
Oct 30 22:45:18.096978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.096941    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:18.153871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.153784    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:18.175428 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.175404    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:18.216592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.216558    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"]
Oct 30 22:45:18.228490 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.228457    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eb4aa1-29890005-rjrlp"]
Oct 30 22:45:18.233155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233124    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca\") pod \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") "
Oct 30 22:45:18.233283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233267    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca\") pod \"81944d12-4147-4747-bd07-baaa00491a29\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") "
Oct 30 22:45:18.233330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233304    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k\" (UniqueName: \"kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k\") pod \"81944d12-4147-4747-bd07-baaa00491a29\" (UID: \"81944d12-4147-4747-bd07-baaa00491a29\") "
Oct 30 22:45:18.233435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233357    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v\" (UniqueName: \"kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v\") pod \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\" (UID: \"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc\") "
Oct 30 22:45:18.233570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233541    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca" pod "43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" (UID: "43cd58ff-0e7d-4146-8357-d93b8c5dc5cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.233736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233672    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca\") pod \"ab923bcb-05ec-462e-a313-9b4308a15d3f\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") "
Oct 30 22:45:18.233832 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233792    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb\" (UniqueName: \"kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb\") pod \"ab923bcb-05ec-462e-a313-9b4308a15d3f\" (UID: \"ab923bcb-05ec-462e-a313-9b4308a15d3f\") "
Oct 30 22:45:18.233905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.233885    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca" pod "81944d12-4147-4747-bd07-baaa00491a29" (UID: "81944d12-4147-4747-bd07-baaa00491a29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.234153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.234114    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca" pod "ab923bcb-05ec-462e-a313-9b4308a15d3f" (UID: "ab923bcb-05ec-462e-a313-9b4308a15d3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.234420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.234399    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.234514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.234427    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81944d12-4147-4747-bd07-baaa00491a29-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.234514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.234441    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab923bcb-05ec-462e-a313-9b4308a15d3f-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.235755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.235729    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v" pod "43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" (UID: "43cd58ff-0e7d-4146-8357-d93b8c5dc5cc"). InnerVolumeSpecName "kube-api-access-rgm6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.236517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.236494    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb" pod "ab923bcb-05ec-462e-a313-9b4308a15d3f" (UID: "ab923bcb-05ec-462e-a313-9b4308a15d3f"). InnerVolumeSpecName "kube-api-access-692vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.236630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.236523    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k" pod "81944d12-4147-4747-bd07-baaa00491a29" (UID: "81944d12-4147-4747-bd07-baaa00491a29"). InnerVolumeSpecName "kube-api-access-vqs9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.335943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.335800    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqs9k\" (UniqueName: \"kubernetes.io/projected/81944d12-4147-4747-bd07-baaa00491a29-kube-api-access-vqs9k\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.335943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.335851    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgm6v\" (UniqueName: \"kubernetes.io/projected/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc-kube-api-access-rgm6v\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.335943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.335866    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-692vb\" (UniqueName: \"kubernetes.io/projected/ab923bcb-05ec-462e-a313-9b4308a15d3f-kube-api-access-692vb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.752771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.752730    5028 scope.go:122] "RemoveContainer" containerID="3933ac9e7bd530a565f15a1f3ee6ba55e511b964494c2ae3113eea1fbcce629b"
Oct 30 22:45:18.765578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.765550    5028 scope.go:122] "RemoveContainer" containerID="fbb13c2d04ce8758c557e7233045b62093f1741c0c42afd77c73dd22c0512464"
Oct 30 22:45:18.775813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.775789    5028 scope.go:122] "RemoveContainer" containerID="0b4aa04809ef35e1a034b2f7683855cf37714bc244df567073439bc429649f83"
Oct 30 22:45:18.791996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.791962    5028 scope.go:122] "RemoveContainer" containerID="74a4505eeea9b8106d826894a6f1ca0c3a1c1f9a9517a9ab0b6750d7ff70bbff"
Oct 30 22:45:18.919143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.919044    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"
Oct 30 22:45:18.919143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.919085    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r" event={"ID":"ab923bcb-05ec-462e-a313-9b4308a15d3f","Type":"ContainerDied","Data":"71e33e68ed512dc9bea886b2c053fbea329f1abb7659d6da7944de734ac983b2"}
Oct 30 22:45:18.919677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.919158    5028 scope.go:122] "RemoveContainer" containerID="e97348130bd0ef59a188f4f22426c0d46adb7dc0b51f80d13d7de19254fdff67"
Oct 30 22:45:18.926112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.924663    5028 generic.go:356] "Generic (PLEG): container finished" podID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerID="a2cdfd056622941721a615db338e1dd31ed10edea13a081c9adc797dc31456f9" exitCode=60
Oct 30 22:45:18.926112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.924919    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerDied","Data":"a2cdfd056622941721a615db338e1dd31ed10edea13a081c9adc797dc31456f9"}
Oct 30 22:45:18.926112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.925356    5028 scope.go:122] "RemoveContainer" containerID="a2cdfd056622941721a615db338e1dd31ed10edea13a081c9adc797dc31456f9"
Oct 30 22:45:18.926112 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:18.925718    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9_odh-ai-gateway-infra(aea0264c-eae4-485c-8d85-bc902d20d7f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6"
Oct 30 22:45:18.928467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.928403    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb" event={"ID":"43cd58ff-0e7d-4146-8357-d93b8c5dc5cc","Type":"ContainerDied","Data":"c5bfbcb8b379a6c6018950ae7b40369b1c1815ca79ab4121beb00508c485f17d"}
Oct 30 22:45:18.928467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.928467    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"
Oct 30 22:45:18.933878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.933323    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"
Oct 30 22:45:18.933878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.933364    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld" event={"ID":"81944d12-4147-4747-bd07-baaa00491a29","Type":"ContainerDied","Data":"83ca7422673da2599899165338425780f97b4006623bf62bda4b498a5ccdf3db"}
Oct 30 22:45:18.942263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.942039    5028 scope.go:122] "RemoveContainer" containerID="7d8869e3d2aef3e4b14b35edb0e67e7f0deaf567a076b3b8e4b1d83c64c52840"
Oct 30 22:45:18.948512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.948330    5028 generic.go:356] "Generic (PLEG): container finished" podID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerID="894999c7400d56a2ae796eb866e1638315caceabda469cc09f0b9db615e5ea88" exitCode=60
Oct 30 22:45:18.948512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.948392    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerDied","Data":"894999c7400d56a2ae796eb866e1638315caceabda469cc09f0b9db615e5ea88"}
Oct 30 22:45:18.948831 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.948812    5028 scope.go:122] "RemoveContainer" containerID="894999c7400d56a2ae796eb866e1638315caceabda469cc09f0b9db615e5ea88"
Oct 30 22:45:18.949138 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:18.949113    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl_odh-ai-gateway-infra(00c418b0-98ab-48f4-858d-a80d0da2da61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61"
Oct 30 22:45:18.951086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.951063    5028 generic.go:356] "Generic (PLEG): container finished" podID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerID="158a0f46e9b054d2f9a8b48490f0c088e103d0d06d096696368263195da60909" exitCode=60
Oct 30 22:45:18.951196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.951107    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerDied","Data":"158a0f46e9b054d2f9a8b48490f0c088e103d0d06d096696368263195da60909"}
Oct 30 22:45:18.951470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.951450    5028 scope.go:122] "RemoveContainer" containerID="158a0f46e9b054d2f9a8b48490f0c088e103d0d06d096696368263195da60909"
Oct 30 22:45:18.951808 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:18.951785    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh_odh-ai-gateway-infra(cacac90f-1c25-4101-a6d3-4fb23a9d406a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a"
Oct 30 22:45:18.960743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.960724    5028 scope.go:122] "RemoveContainer" containerID="e68af4908900b3a5a1caa636ad53443d1ebebb127c0243f91cc45ecfbae4e86f"
Oct 30 22:45:18.974184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.974162    5028 scope.go:122] "RemoveContainer" containerID="7d749c86eab6f436af450bcb9ea416cd45d5167c7f411b516cc40627875c076a"
Oct 30 22:45:18.983358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.983334    5028 scope.go:122] "RemoveContainer" containerID="777a9806a8c66bc79254c300da64e86f7678b1f90ddc95f9b03277511642a726"
Oct 30 22:45:18.996643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:18.996620    5028 scope.go:122] "RemoveContainer" containerID="1f86b2d2f5b9c3575dfc0899a9dba6d4b0ea7bdeed6778ff1d5146efa0e03529"
Oct 30 22:45:19.116059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.115945    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"]
Oct 30 22:45:19.124323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.124288    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"]
Oct 30 22:45:19.130171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.130137    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d585d2-29890005-jvdnb"]
Oct 30 22:45:19.228971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.228936    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"]
Oct 30 22:45:19.233253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.233223    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb3406-29890005-88lld"]
Oct 30 22:45:19.292969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.292929    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" path="/var/lib/kubelet/pods/43cd58ff-0e7d-4146-8357-d93b8c5dc5cc/volumes"
Oct 30 22:45:19.293489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.293467    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" path="/var/lib/kubelet/pods/6d3791f8-4e73-4524-a52f-36ad674307b5/volumes"
Oct 30 22:45:19.294121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.294100    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81944d12-4147-4747-bd07-baaa00491a29" path="/var/lib/kubelet/pods/81944d12-4147-4747-bd07-baaa00491a29/volumes"
Oct 30 22:45:19.336733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.336702    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"]
Oct 30 22:45:19.345814 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.345786    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29890005-ht86r"]
Oct 30 22:45:19.351253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.351227    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"]
Oct 30 22:45:19.680322 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:19.680227    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"]
Oct 30 22:45:20.158868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.158837    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:20.206431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.206404    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:20.226540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.226478    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:20.268068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.268033    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2\" (UniqueName: \"kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2\") pod \"aea0264c-eae4-485c-8d85-bc902d20d7f6\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") "
Oct 30 22:45:20.268264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.268115    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca\") pod \"aea0264c-eae4-485c-8d85-bc902d20d7f6\" (UID: \"aea0264c-eae4-485c-8d85-bc902d20d7f6\") "
Oct 30 22:45:20.268543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.268518    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca" pod "aea0264c-eae4-485c-8d85-bc902d20d7f6" (UID: "aea0264c-eae4-485c-8d85-bc902d20d7f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.268625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.268597    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea0264c-eae4-485c-8d85-bc902d20d7f6-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.270497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.270472    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2" pod "aea0264c-eae4-485c-8d85-bc902d20d7f6" (UID: "aea0264c-eae4-485c-8d85-bc902d20d7f6"). InnerVolumeSpecName "kube-api-access-z7tl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.369425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.369393    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx\" (UniqueName: \"kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx\") pod \"00c418b0-98ab-48f4-858d-a80d0da2da61\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") "
Oct 30 22:45:20.369646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.369519    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn\" (UniqueName: \"kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn\") pod \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") "
Oct 30 22:45:20.369751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.369698    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca\") pod \"00c418b0-98ab-48f4-858d-a80d0da2da61\" (UID: \"00c418b0-98ab-48f4-858d-a80d0da2da61\") "
Oct 30 22:45:20.369751 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.369740    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca\") pod \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\" (UID: \"cacac90f-1c25-4101-a6d3-4fb23a9d406a\") "
Oct 30 22:45:20.370126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.370093    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca" pod "cacac90f-1c25-4101-a6d3-4fb23a9d406a" (UID: "cacac90f-1c25-4101-a6d3-4fb23a9d406a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.370126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.370110    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca" pod "00c418b0-98ab-48f4-858d-a80d0da2da61" (UID: "00c418b0-98ab-48f4-858d-a80d0da2da61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.370811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.370781    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7tl2\" (UniqueName: \"kubernetes.io/projected/aea0264c-eae4-485c-8d85-bc902d20d7f6-kube-api-access-z7tl2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.370904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.370813    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c418b0-98ab-48f4-858d-a80d0da2da61-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.370904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.370830    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cacac90f-1c25-4101-a6d3-4fb23a9d406a-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.372312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.372282    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx" pod "00c418b0-98ab-48f4-858d-a80d0da2da61" (UID: "00c418b0-98ab-48f4-858d-a80d0da2da61"). InnerVolumeSpecName "kube-api-access-fwfsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.372407 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.372348    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn" pod "cacac90f-1c25-4101-a6d3-4fb23a9d406a" (UID: "cacac90f-1c25-4101-a6d3-4fb23a9d406a"). InnerVolumeSpecName "kube-api-access-ltkpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.472415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.472373    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfsx\" (UniqueName: \"kubernetes.io/projected/00c418b0-98ab-48f4-858d-a80d0da2da61-kube-api-access-fwfsx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.472415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.472408    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltkpn\" (UniqueName: \"kubernetes.io/projected/cacac90f-1c25-4101-a6d3-4fb23a9d406a-kube-api-access-ltkpn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.979867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.979842    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"
Oct 30 22:45:20.980073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.979920    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl" event={"ID":"00c418b0-98ab-48f4-858d-a80d0da2da61","Type":"ContainerDied","Data":"f10ad5b30c471d845c6de0c01d20f4fdf3ba30bcddd654f84d84b9be6aae32be"}
Oct 30 22:45:20.980073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.979989    5028 scope.go:122] "RemoveContainer" containerID="894999c7400d56a2ae796eb866e1638315caceabda469cc09f0b9db615e5ea88"
Oct 30 22:45:20.981629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.981601    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"
Oct 30 22:45:20.981782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.981635    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh" event={"ID":"cacac90f-1c25-4101-a6d3-4fb23a9d406a","Type":"ContainerDied","Data":"9a7e144aec361b2af88e8e174b4b78dcb260b661aaee5a78c6e55b772d475c87"}
Oct 30 22:45:20.983259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.983234    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9" event={"ID":"aea0264c-eae4-485c-8d85-bc902d20d7f6","Type":"ContainerDied","Data":"05434d23497e8e4286ed3509ddb5b36db584210fff27f7f97d5e00c83e7f7931"}
Oct 30 22:45:20.983389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.983271    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"
Oct 30 22:45:20.992172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:20.992150    5028 scope.go:122] "RemoveContainer" containerID="158a0f46e9b054d2f9a8b48490f0c088e103d0d06d096696368263195da60909"
Oct 30 22:45:21.005034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.005014    5028 scope.go:122] "RemoveContainer" containerID="a2cdfd056622941721a615db338e1dd31ed10edea13a081c9adc797dc31456f9"
Oct 30 22:45:21.011261 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:21.011230    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacac90f_1c25_4101_a6d3_4fb23a9d406a.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:45:21.139714 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.139665    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"]
Oct 30 22:45:21.187063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.187032    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9fce61-29890005-xlxc9"]
Oct 30 22:45:21.250318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.250240    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"]
Oct 30 22:45:21.253506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.253481    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d6b55f-29890005-htzcl"]
Oct 30 22:45:21.292100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.292066    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" path="/var/lib/kubelet/pods/00c418b0-98ab-48f4-858d-a80d0da2da61/volumes"
Oct 30 22:45:21.292545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.292529    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" path="/var/lib/kubelet/pods/ab923bcb-05ec-462e-a313-9b4308a15d3f/volumes"
Oct 30 22:45:21.293162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.293141    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" path="/var/lib/kubelet/pods/aea0264c-eae4-485c-8d85-bc902d20d7f6/volumes"
Oct 30 22:45:21.358437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.358400    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"]
Oct 30 22:45:21.368664 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.368629    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29890005-kl7fh"]
Oct 30 22:45:21.953534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:21.953500    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:23.295710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.294258    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" path="/var/lib/kubelet/pods/cacac90f-1c25-4101-a6d3-4fb23a9d406a/volumes"
Oct 30 22:45:23.323995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.323966    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"]
Oct 30 22:45:23.324672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324648    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:23.324672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324672    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324705    5028 cpu_manager.go:414] "containerMap: removing container" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324714    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324729    5028 cpu_manager.go:414] "containerMap: removing container" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324736    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324759    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324766    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324775    5028 cpu_manager.go:414] "containerMap: removing container" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324782    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324800    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324807    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324818    5028 cpu_manager.go:414] "containerMap: removing container" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324825    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324834    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.324842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324840    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324849    5028 cpu_manager.go:414] "containerMap: removing container" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324855    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324869    5028 cpu_manager.go:414] "containerMap: removing container" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324876    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324888    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324895    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324910    5028 cpu_manager.go:414] "containerMap: removing container" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324914    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324927    5028 cpu_manager.go:414] "containerMap: removing container" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324931    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324937    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324941    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324949    5028 cpu_manager.go:414] "containerMap: removing container" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324955    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324966    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324970    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324979    5028 cpu_manager.go:414] "containerMap: removing container" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.324983    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325112    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325120    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325126    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325133    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325139    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325145    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325149    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325156    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325161    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325166    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325172    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325177    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325184    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c418b0-98ab-48f4-858d-a80d0da2da61" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325189    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:23.325279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.325195    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:23.330165 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.330144    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.382769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.382737    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"]
Oct 30 22:45:23.424576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.424541    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.424804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.424736    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.424804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.424795    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhw8r\" (UniqueName: \"kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.425566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.425541    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:23.425566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.425552    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:23.425786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.425574    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:23.429101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.429084    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:45:23.526670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.526634    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.526889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.526721    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhw8r\" (UniqueName: \"kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.526889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.526855    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.527352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.527331    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.529287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.529259    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.586232 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.586154    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhw8r\" (UniqueName: \"kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r\") pod \"maas-api-revoke-sub-cec3a832-5kp5c\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:23.648410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:23.648376    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:24.023241 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:24.023183    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"]
Oct 30 22:45:24.023377 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:24.023203    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052aae37_53a3_4ea1_86a7_ca267534e209.slice/crio-022e64a3910c53972f4f73ccb58127cc3f2cd927f1fd74d373a48da89a2b3ac6 WatchSource:0}: Error finding container 022e64a3910c53972f4f73ccb58127cc3f2cd927f1fd74d373a48da89a2b3ac6: Status 404 returned error can't find the container with id 022e64a3910c53972f4f73ccb58127cc3f2cd927f1fd74d373a48da89a2b3ac6
Oct 30 22:45:24.038570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:24.038538    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c" event={"ID":"052aae37-53a3-4ea1-86a7-ca267534e209","Type":"ContainerStarted","Data":"022e64a3910c53972f4f73ccb58127cc3f2cd927f1fd74d373a48da89a2b3ac6"}
Oct 30 22:45:25.050086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:25.050044    5028 generic.go:356] "Generic (PLEG): container finished" podID="052aae37-53a3-4ea1-86a7-ca267534e209" containerID="c0c3aeecb48304c327789641ced45271fbd7abff149e35051ff7e775ef424568" exitCode=0
Oct 30 22:45:25.050557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:25.050090    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c" event={"ID":"052aae37-53a3-4ea1-86a7-ca267534e209","Type":"ContainerDied","Data":"c0c3aeecb48304c327789641ced45271fbd7abff149e35051ff7e775ef424568"}
Oct 30 22:45:26.232734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.232704    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:26.257906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.257871    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca\") pod \"052aae37-53a3-4ea1-86a7-ca267534e209\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") "
Oct 30 22:45:26.258088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.257944    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca\") pod \"052aae37-53a3-4ea1-86a7-ca267534e209\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") "
Oct 30 22:45:26.258088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.258004    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r\" (UniqueName: \"kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r\") pod \"052aae37-53a3-4ea1-86a7-ca267534e209\" (UID: \"052aae37-53a3-4ea1-86a7-ca267534e209\") "
Oct 30 22:45:26.258276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.258252    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca" pod "052aae37-53a3-4ea1-86a7-ca267534e209" (UID: "052aae37-53a3-4ea1-86a7-ca267534e209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:26.258515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.258497    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:26.260246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.260218    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca" pod "052aae37-53a3-4ea1-86a7-ca267534e209" (UID: "052aae37-53a3-4ea1-86a7-ca267534e209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:26.260246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.260235    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r" pod "052aae37-53a3-4ea1-86a7-ca267534e209" (UID: "052aae37-53a3-4ea1-86a7-ca267534e209"). InnerVolumeSpecName "kube-api-access-dhw8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:26.359576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.359497    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052aae37-53a3-4ea1-86a7-ca267534e209-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:26.359576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.359529    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhw8r\" (UniqueName: \"kubernetes.io/projected/052aae37-53a3-4ea1-86a7-ca267534e209-kube-api-access-dhw8r\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:26.455488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:26.455451    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:27.068213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:27.068183    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"
Oct 30 22:45:27.068402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:27.068214    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c" event={"ID":"052aae37-53a3-4ea1-86a7-ca267534e209","Type":"ContainerDied","Data":"022e64a3910c53972f4f73ccb58127cc3f2cd927f1fd74d373a48da89a2b3ac6"}
Oct 30 22:45:27.068402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:27.068261    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="022e64a3910c53972f4f73ccb58127cc3f2cd927f1fd74d373a48da89a2b3ac6"
Oct 30 22:45:28.436962 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:28.436925    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:45:29.312759 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.312724    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"]
Oct 30 22:45:29.312933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.312912    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" podUID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" containerName="discovery" containerID="cri-o://1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c" gracePeriod=30
Oct 30 22:45:29.623069 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.623034    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:45:29.702877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.702843    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.702944    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.702993    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.703040    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.703254    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.703299    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.703333    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq\") pod \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\" (UID: \"27ed34d1-04a4-4d50-86ba-b9dbfce6246f\") "
Oct 30 22:45:29.703350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.703338    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:29.703995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.703938    5028 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-ca-configmap\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.705402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.705360    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:29.705517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.705498    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:29.705724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.705669    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:29.705848 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.705828    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:29.705958 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.705940    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:29.706049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.706029    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq" pod "27ed34d1-04a4-4d50-86ba-b9dbfce6246f" (UID: "27ed34d1-04a4-4d50-86ba-b9dbfce6246f"). InnerVolumeSpecName "kube-api-access-dpxbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:29.804914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.804880    5028 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-local-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.804914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.804909    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.804914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.804918    5028 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-cacerts\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.805167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.804926    5028 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-kubeconfig\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.805167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.804936    5028 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-istio-csr-dns-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.805167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:29.804946    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxbq\" (UniqueName: \"kubernetes.io/projected/27ed34d1-04a4-4d50-86ba-b9dbfce6246f-kube-api-access-dpxbq\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:30.092051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.092012    5028 generic.go:356] "Generic (PLEG): container finished" podID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" containerID="1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c" exitCode=0
Oct 30 22:45:30.092263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.092059    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"
Oct 30 22:45:30.092263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.092091    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" event={"ID":"27ed34d1-04a4-4d50-86ba-b9dbfce6246f","Type":"ContainerDied","Data":"1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c"}
Oct 30 22:45:30.092263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.092160    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b" event={"ID":"27ed34d1-04a4-4d50-86ba-b9dbfce6246f","Type":"ContainerDied","Data":"f0b4285017939b32988fead6b5fdd29c00b0fd9a6875007e137b155b0187df03"}
Oct 30 22:45:30.092263 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.092181    5028 scope.go:122] "RemoveContainer" containerID="1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c"
Oct 30 22:45:30.104888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.104862    5028 scope.go:122] "RemoveContainer" containerID="1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c"
Oct 30 22:45:30.105157 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:30.105137    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c\": container with ID starting with 1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c not found: ID does not exist" containerID="1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c"
Oct 30 22:45:30.105216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.105166    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c"} err="failed to get container status \"1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c\": rpc error: code = NotFound desc = could not find container \"1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c\": container with ID starting with 1e409c1311b6a1ec699f921c770005c61e797bdd90f351b0e53e31cb83fe742c not found: ID does not exist"
Oct 30 22:45:30.251268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.251230    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"]
Oct 30 22:45:30.298208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.298160    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qjq6b"]
Oct 30 22:45:30.945001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:30.944965    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:31.292259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:31.292228    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" path="/var/lib/kubelet/pods/27ed34d1-04a4-4d50-86ba-b9dbfce6246f/volumes"
Oct 30 22:45:34.884573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:34.884532    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.426668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:35.426625    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:36.348060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.348027    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:36.383221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383182    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"]
Oct 30 22:45:36.383882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383860    5028 cpu_manager.go:414] "containerMap: removing container" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:36.383882 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383882    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383907    5028 cpu_manager.go:414] "containerMap: removing container" podUID="052aae37-53a3-4ea1-86a7-ca267534e209" containerName="revoke-keys"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383914    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052aae37-53a3-4ea1-86a7-ca267534e209" containerName="revoke-keys"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383942    5028 cpu_manager.go:414] "containerMap: removing container" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383951    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383965    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383972    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383992    5028 cpu_manager.go:414] "containerMap: removing container" podUID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" containerName="discovery"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.383999    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" containerName="discovery"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384027    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:36.384082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384034    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384210    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="052aae37-53a3-4ea1-86a7-ca267534e209" containerName="revoke-keys"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384228    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacac90f-1c25-4101-a6d3-4fb23a9d406a" containerName="cleanup"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384237    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab923bcb-05ec-462e-a313-9b4308a15d3f" containerName="cleanup"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384249    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ed34d1-04a4-4d50-86ba-b9dbfce6246f" containerName="discovery"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384261    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea0264c-eae4-485c-8d85-bc902d20d7f6" containerName="cleanup"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384272    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3791f8-4e73-4524-a52f-36ad674307b5" containerName="cleanup"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384282    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="81944d12-4147-4747-bd07-baaa00491a29" containerName="cleanup"
Oct 30 22:45:36.384607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.384291    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cd58ff-0e7d-4146-8357-d93b8c5dc5cc" containerName="cleanup"
Oct 30 22:45:36.387999 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.387978    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.444185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.444152    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"]
Oct 30 22:45:36.485613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.485580    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:36.485899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.485603    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:36.486267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.485631    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:36.486267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.485676    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:45:36.500291 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.500262    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzbt\" (UniqueName: \"kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.500465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.500333    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.500465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.500419    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.553005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.552968    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:36.602108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.602018    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzbt\" (UniqueName: \"kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.602108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.602098    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.602356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.602135    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.602933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.602907    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.604512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.604486    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.661102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.661070    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzbt\" (UniqueName: \"kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt\") pod \"maas-api-revoke-sub-e240c47a-dljhs\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.701656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.701624    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:36.709869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.709842    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:36.917341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.917240    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"]
Oct 30 22:45:36.920713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:36.920674    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9fce61-bgkqh"]
Oct 30 22:45:37.074126 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:37.074093    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"]
Oct 30 22:45:37.074411 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:37.074387    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f0990d_b0b9_4c14_8916_0d6ccb109431.slice/crio-b691d9e45fe7b3d2cfe7ce7fb847a3fb69af6208e2880f86a4993aa0ef3d90b4 WatchSource:0}: Error finding container b691d9e45fe7b3d2cfe7ce7fb847a3fb69af6208e2880f86a4993aa0ef3d90b4: Status 404 returned error can't find the container with id b691d9e45fe7b3d2cfe7ce7fb847a3fb69af6208e2880f86a4993aa0ef3d90b4
Oct 30 22:45:37.148950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:37.148914    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs" event={"ID":"f1f0990d-b0b9-4c14-8916-0d6ccb109431","Type":"ContainerStarted","Data":"b691d9e45fe7b3d2cfe7ce7fb847a3fb69af6208e2880f86a4993aa0ef3d90b4"}
Oct 30 22:45:37.291297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:37.291259    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4f38ca-4642-4454-aa9d-24f7641b6874" path="/var/lib/kubelet/pods/ef4f38ca-4642-4454-aa9d-24f7641b6874/volumes"
Oct 30 22:45:37.365857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:37.365825    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:38.159200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:38.159112    5028 generic.go:356] "Generic (PLEG): container finished" podID="f1f0990d-b0b9-4c14-8916-0d6ccb109431" containerID="8bb0f9298ff0c23596a195a555678033ad93dbf8c8c0f4147dc9f5758ef03b93" exitCode=0
Oct 30 22:45:38.159383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:38.159195    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs" event={"ID":"f1f0990d-b0b9-4c14-8916-0d6ccb109431","Type":"ContainerDied","Data":"8bb0f9298ff0c23596a195a555678033ad93dbf8c8c0f4147dc9f5758ef03b93"}
Oct 30 22:45:39.315247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.315219    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:39.439183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.439088    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca\") pod \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") "
Oct 30 22:45:39.439371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.439226    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt\" (UniqueName: \"kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt\") pod \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") "
Oct 30 22:45:39.439371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.439336    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca\") pod \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\" (UID: \"f1f0990d-b0b9-4c14-8916-0d6ccb109431\") "
Oct 30 22:45:39.439890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.439858    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca" pod "f1f0990d-b0b9-4c14-8916-0d6ccb109431" (UID: "f1f0990d-b0b9-4c14-8916-0d6ccb109431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:39.441556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.441528    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca" pod "f1f0990d-b0b9-4c14-8916-0d6ccb109431" (UID: "f1f0990d-b0b9-4c14-8916-0d6ccb109431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:39.441556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.441546    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt" pod "f1f0990d-b0b9-4c14-8916-0d6ccb109431" (UID: "f1f0990d-b0b9-4c14-8916-0d6ccb109431"). InnerVolumeSpecName "kube-api-access-7nzbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:39.541213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.541179    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nzbt\" (UniqueName: \"kubernetes.io/projected/f1f0990d-b0b9-4c14-8916-0d6ccb109431-kube-api-access-7nzbt\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:39.541213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.541208    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:39.541213 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:39.541218    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f0990d-b0b9-4c14-8916-0d6ccb109431-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.175350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:40.175320    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"
Oct 30 22:45:40.175350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:40.175333    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs" event={"ID":"f1f0990d-b0b9-4c14-8916-0d6ccb109431","Type":"ContainerDied","Data":"b691d9e45fe7b3d2cfe7ce7fb847a3fb69af6208e2880f86a4993aa0ef3d90b4"}
Oct 30 22:45:40.175574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:40.175380    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b691d9e45fe7b3d2cfe7ce7fb847a3fb69af6208e2880f86a4993aa0ef3d90b4"
Oct 30 22:45:41.367749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:41.367712    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.672437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:43.672401    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.870167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:43.870135    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.057042 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:44.057008    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:44.523621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:44.523568    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:45.028209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.028168    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"]
Oct 30 22:45:45.028794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.028777    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f1f0990d-b0b9-4c14-8916-0d6ccb109431" containerName="revoke-keys"
Oct 30 22:45:45.028794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.028793    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f0990d-b0b9-4c14-8916-0d6ccb109431" containerName="revoke-keys"
Oct 30 22:45:45.028939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.028889    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f0990d-b0b9-4c14-8916-0d6ccb109431" containerName="revoke-keys"
Oct 30 22:45:45.032593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.032566    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.087623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.087583    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"]
Oct 30 22:45:45.129261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.129217    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:45.129437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.129317    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:45.129437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.129322    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:45.135020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.134995    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:45:45.215706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.215661    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.215935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.215829    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.215935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.215898    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvsw\" (UniqueName: \"kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.317303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.317211    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.317478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.317350    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvsw\" (UniqueName: \"kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.317478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.317447    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.317901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.317879    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.319924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.319899    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.376954 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.376922    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvsw\" (UniqueName: \"kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw\") pod \"maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:45.646459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:45.646376    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:46.023257 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:45:46.023217    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543a5569_ca67_438c_befd_a693fd7c7a08.slice/crio-e1933175b58e51230ff4d430737415bd3659da830a4f063979219f5cd9bb1056 WatchSource:0}: Error finding container e1933175b58e51230ff4d430737415bd3659da830a4f063979219f5cd9bb1056: Status 404 returned error can't find the container with id e1933175b58e51230ff4d430737415bd3659da830a4f063979219f5cd9bb1056
Oct 30 22:45:46.024046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:46.023957    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"]
Oct 30 22:45:46.140744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:46.140714    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.228221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:46.228185    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs" event={"ID":"543a5569-ca67-438c-befd-a693fd7c7a08","Type":"ContainerStarted","Data":"e1933175b58e51230ff4d430737415bd3659da830a4f063979219f5cd9bb1056"}
Oct 30 22:45:47.237643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:47.237608    5028 generic.go:356] "Generic (PLEG): container finished" podID="543a5569-ca67-438c-befd-a693fd7c7a08" containerID="274bbbfe6f1fe60b285117dc8f2b071b480050b35d9a36a560ab8e78e30173a1" exitCode=0
Oct 30 22:45:47.238077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:47.237723    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs" event={"ID":"543a5569-ca67-438c-befd-a693fd7c7a08","Type":"ContainerDied","Data":"274bbbfe6f1fe60b285117dc8f2b071b480050b35d9a36a560ab8e78e30173a1"}
Oct 30 22:45:48.417806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.417775    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:48.556253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.556157    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca\") pod \"543a5569-ca67-438c-befd-a693fd7c7a08\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") "
Oct 30 22:45:48.556435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.556296    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw\" (UniqueName: \"kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw\") pod \"543a5569-ca67-438c-befd-a693fd7c7a08\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") "
Oct 30 22:45:48.556510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.556454    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca\") pod \"543a5569-ca67-438c-befd-a693fd7c7a08\" (UID: \"543a5569-ca67-438c-befd-a693fd7c7a08\") "
Oct 30 22:45:48.556933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.556902    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca" pod "543a5569-ca67-438c-befd-a693fd7c7a08" (UID: "543a5569-ca67-438c-befd-a693fd7c7a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:48.557097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.556992    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.558680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.558651    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw" pod "543a5569-ca67-438c-befd-a693fd7c7a08" (UID: "543a5569-ca67-438c-befd-a693fd7c7a08"). InnerVolumeSpecName "kube-api-access-nvvsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:48.558965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.558933    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca" pod "543a5569-ca67-438c-befd-a693fd7c7a08" (UID: "543a5569-ca67-438c-befd-a693fd7c7a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:48.658522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.658483    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvsw\" (UniqueName: \"kubernetes.io/projected/543a5569-ca67-438c-befd-a693fd7c7a08-kube-api-access-nvvsw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.658522 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:48.658519    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543a5569-ca67-438c-befd-a693fd7c7a08-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:49.266487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:49.266447    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"
Oct 30 22:45:49.266487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:49.266453    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs" event={"ID":"543a5569-ca67-438c-befd-a693fd7c7a08","Type":"ContainerDied","Data":"e1933175b58e51230ff4d430737415bd3659da830a4f063979219f5cd9bb1056"}
Oct 30 22:45:49.266752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:49.266542    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1933175b58e51230ff4d430737415bd3659da830a4f063979219f5cd9bb1056"
Oct 30 22:45:49.470656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:49.470615    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"]
Oct 30 22:45:49.592062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:49.591974    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:50.997530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:50.997497    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.144730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:52.144645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:55.342536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:55.342490    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:57.693304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:57.693274    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:57.850605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:57.850567    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:58.436453 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:45:58.436412    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:45:59.847226 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:45:59.847194    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:01.318308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.318269    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:01.826181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.826143    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"]
Oct 30 22:46:01.826789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.826775    5028 cpu_manager.go:414] "containerMap: removing container" podUID="543a5569-ca67-438c-befd-a693fd7c7a08" containerName="revoke-keys"
Oct 30 22:46:01.826843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.826790    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543a5569-ca67-438c-befd-a693fd7c7a08" containerName="revoke-keys"
Oct 30 22:46:01.826918 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.826908    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="543a5569-ca67-438c-befd-a693fd7c7a08" containerName="revoke-keys"
Oct 30 22:46:01.830790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.830767    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:01.886541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.886508    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"]
Oct 30 22:46:01.918386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.918336    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6kjx\" (UniqueName: \"kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:01.918601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.918395    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:01.918670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.918611    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:01.928469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.928440    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:01.928469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.928438    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:01.928693 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.928486    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:01.928693 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:01.928566    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:46:02.019946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.019902    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6kjx\" (UniqueName: \"kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.020167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.019961    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.020167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.020075    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.020850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.020828    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.022641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.022609    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.079563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.079478    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6kjx\" (UniqueName: \"kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx\") pod \"maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.149443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.149407    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:02.531186 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:02.531156    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"]
Oct 30 22:46:02.531757 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:46:02.531732    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf477695a_21ec_4127_92e0_57e4858252d8.slice/crio-3af641037aaf6ed30c33e7174b1feef1d5bbe5adc3c949df00561991fd7b6579 WatchSource:0}: Error finding container 3af641037aaf6ed30c33e7174b1feef1d5bbe5adc3c949df00561991fd7b6579: Status 404 returned error can't find the container with id 3af641037aaf6ed30c33e7174b1feef1d5bbe5adc3c949df00561991fd7b6579
Oct 30 22:46:03.376233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:03.376192    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6" event={"ID":"f477695a-21ec-4127-92e0-57e4858252d8","Type":"ContainerStarted","Data":"8989a5ee30cc9be641978f5da8406e2c0fb6e6607e7b505d49091784b378c871"}
Oct 30 22:46:03.376402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:03.376254    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6" event={"ID":"f477695a-21ec-4127-92e0-57e4858252d8","Type":"ContainerStarted","Data":"3af641037aaf6ed30c33e7174b1feef1d5bbe5adc3c949df00561991fd7b6579"}
Oct 30 22:46:03.488372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:03.488309    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6" podStartSLOduration=2.48829132 podStartE2EDuration="2.48829132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:03.486897249 +0000 UTC m=+2694.336410225" watchObservedRunningTime="2026-10-30 22:46:03.48829132 +0000 UTC m=+2694.337804315"
Oct 30 22:46:04.142585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.142547    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"]
Oct 30 22:46:04.147109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.147087    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.202100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.202063    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"]
Oct 30 22:46:04.206622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.206595    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"]
Oct 30 22:46:04.206811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.206747    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.249195 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.249156    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.249392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.249203    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbvl\" (UniqueName: \"kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.249392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.249234    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.249392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.249370    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.249571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.249426    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.249571 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.249509    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lckv\" (UniqueName: \"kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.265527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.265493    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"]
Oct 30 22:46:04.350425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.350386    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.350425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.350428    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbvl\" (UniqueName: \"kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.350734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.350446    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.350734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.350484    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.350734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.350519    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.350734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.350537    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lckv\" (UniqueName: \"kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.351302 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.351272    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.351448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.351272    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.353221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.353185    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.353341 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.353185    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.385756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.385725    5028 generic.go:356] "Generic (PLEG): container finished" podID="f477695a-21ec-4127-92e0-57e4858252d8" containerID="8989a5ee30cc9be641978f5da8406e2c0fb6e6607e7b505d49091784b378c871" exitCode=0
Oct 30 22:46:04.385920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.385810    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6" event={"ID":"f477695a-21ec-4127-92e0-57e4858252d8","Type":"ContainerDied","Data":"8989a5ee30cc9be641978f5da8406e2c0fb6e6607e7b505d49091784b378c871"}
Oct 30 22:46:04.409314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.409249    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lckv\" (UniqueName: \"kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv\") pod \"maas-api-revoke-sub-358ebcb9-qk2fc\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.409467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.409360    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbvl\" (UniqueName: \"kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl\") pod \"maas-api-revoke-sub-c6543fbd-mfqh4\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.462844 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.462805    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:04.492524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.492483    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" containerID="cri-o://94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4" gracePeriod=45
Oct 30 22:46:04.524175 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.523909    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:04.665203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.665106    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": dial tcp 10.132.0.108:8000: connect: connection refused"
Oct 30 22:46:04.807140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.807108    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:46:04.842645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.842615    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"]
Oct 30 22:46:04.842808 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:46:04.842671    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b606d7_09d8_4a9d_97a5_6697722ead44.slice/crio-a384c95a115e175733ea38c961c73c049017b6b215cd88ca774ab154b43a8622 WatchSource:0}: Error finding container a384c95a115e175733ea38c961c73c049017b6b215cd88ca774ab154b43a8622: Status 404 returned error can't find the container with id a384c95a115e175733ea38c961c73c049017b6b215cd88ca774ab154b43a8622
Oct 30 22:46:04.857822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.857792    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b\" (UniqueName: \"kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b\") pod \"44b42215-85eb-46a2-9951-944feed99b07\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") "
Oct 30 22:46:04.857946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.857856    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir\") pod \"44b42215-85eb-46a2-9951-944feed99b07\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") "
Oct 30 22:46:04.857946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.857883    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs\" (UniqueName: \"kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs\") pod \"44b42215-85eb-46a2-9951-944feed99b07\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") "
Oct 30 22:46:04.857946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.857914    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home\") pod \"44b42215-85eb-46a2-9951-944feed99b07\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") "
Oct 30 22:46:04.858142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.857962    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm\") pod \"44b42215-85eb-46a2-9951-944feed99b07\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") "
Oct 30 22:46:04.858142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858036    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache\") pod \"44b42215-85eb-46a2-9951-944feed99b07\" (UID: \"44b42215-85eb-46a2-9951-944feed99b07\") "
Oct 30 22:46:04.858142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858123    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir" pod "44b42215-85eb-46a2-9951-944feed99b07" (UID: "44b42215-85eb-46a2-9951-944feed99b07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:04.858343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858296    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home" pod "44b42215-85eb-46a2-9951-944feed99b07" (UID: "44b42215-85eb-46a2-9951-944feed99b07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:04.858551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858528    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache" pod "44b42215-85eb-46a2-9951-944feed99b07" (UID: "44b42215-85eb-46a2-9951-944feed99b07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:04.858724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858682    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:04.858724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858720    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:04.858891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.858733    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:04.860562 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.860534    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm" pod "44b42215-85eb-46a2-9951-944feed99b07" (UID: "44b42215-85eb-46a2-9951-944feed99b07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:04.860663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.860580    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b" pod "44b42215-85eb-46a2-9951-944feed99b07" (UID: "44b42215-85eb-46a2-9951-944feed99b07"). InnerVolumeSpecName "kube-api-access-4mh2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:04.860796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.860775    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs" pod "44b42215-85eb-46a2-9951-944feed99b07" (UID: "44b42215-85eb-46a2-9951-944feed99b07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:04.944574 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.944540    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"]
Oct 30 22:46:04.960021 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.959982    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mh2b\" (UniqueName: \"kubernetes.io/projected/44b42215-85eb-46a2-9951-944feed99b07-kube-api-access-4mh2b\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:04.960181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.960026    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b42215-85eb-46a2-9951-944feed99b07-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:04.960181 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.960043    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b42215-85eb-46a2-9951-944feed99b07-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:04.980591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:04.980552    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.019533 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:46:05.019490    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b1a0d_e240_49b9_9eb2_2127e876a5c8.slice/crio-6006ab9b449621e8103444932f65cecc0b5f4613af4d336d7d591b28d8eff71e WatchSource:0}: Error finding container 6006ab9b449621e8103444932f65cecc0b5f4613af4d336d7d591b28d8eff71e: Status 404 returned error can't find the container with id 6006ab9b449621e8103444932f65cecc0b5f4613af4d336d7d591b28d8eff71e
Oct 30 22:46:05.395647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.395611    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4" event={"ID":"78b606d7-09d8-4a9d-97a5-6697722ead44","Type":"ContainerStarted","Data":"10e229ba72c9ebae851aa9c06e7810da0fb059a99bdb6ddab3e5386a3e6acc63"}
Oct 30 22:46:05.396091 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.395670    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4" event={"ID":"78b606d7-09d8-4a9d-97a5-6697722ead44","Type":"ContainerStarted","Data":"a384c95a115e175733ea38c961c73c049017b6b215cd88ca774ab154b43a8622"}
Oct 30 22:46:05.397077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.397052    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc" event={"ID":"d56b1a0d-e240-49b9-9eb2-2127e876a5c8","Type":"ContainerStarted","Data":"6006ab9b449621e8103444932f65cecc0b5f4613af4d336d7d591b28d8eff71e"}
Oct 30 22:46:05.398352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.398331    5028 generic.go:356] "Generic (PLEG): container finished" podID="44b42215-85eb-46a2-9951-944feed99b07" containerID="94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4" exitCode=0
Oct 30 22:46:05.398439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.398369    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"
Oct 30 22:46:05.398439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.398416    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" event={"ID":"44b42215-85eb-46a2-9951-944feed99b07","Type":"ContainerDied","Data":"94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4"}
Oct 30 22:46:05.398549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.398459    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84" event={"ID":"44b42215-85eb-46a2-9951-944feed99b07","Type":"ContainerDied","Data":"f79a7a8972c05407456b3f34d6a741abaceab794828b1022097f016f3915479f"}
Oct 30 22:46:05.398549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.398486    5028 scope.go:122] "RemoveContainer" containerID="94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4"
Oct 30 22:46:05.409800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.409779    5028 scope.go:122] "RemoveContainer" containerID="94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4"
Oct 30 22:46:05.410129 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:46:05.410104    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4\": container with ID starting with 94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4 not found: ID does not exist" containerID="94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4"
Oct 30 22:46:05.410222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.410141    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4"} err="failed to get container status \"94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4\": rpc error: code = NotFound desc = could not find container \"94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4\": container with ID starting with 94f5b148de38958a4c41483f75ccba1041cd4063661b28b1d8bd1b504122eef4 not found: ID does not exist"
Oct 30 22:46:05.510426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.510359    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4" podStartSLOduration=1.5103400219999998 podStartE2EDuration="1.510340022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:05.507293007 +0000 UTC m=+2696.356805982" watchObservedRunningTime="2026-10-30 22:46:05.510340022 +0000 UTC m=+2696.359852996"
Oct 30 22:46:05.553767 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.553677    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:05.615610 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.615573    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"]
Oct 30 22:46:05.619363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.619339    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-w3-23f3d4-kserve-5447b767c4-plp84"]
Oct 30 22:46:05.669390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.669355    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca\") pod \"f477695a-21ec-4127-92e0-57e4858252d8\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") "
Oct 30 22:46:05.669725 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.669680    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx\" (UniqueName: \"kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx\") pod \"f477695a-21ec-4127-92e0-57e4858252d8\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") "
Oct 30 22:46:05.670227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.669774    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca" pod "f477695a-21ec-4127-92e0-57e4858252d8" (UID: "f477695a-21ec-4127-92e0-57e4858252d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:05.670227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.669939    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca\") pod \"f477695a-21ec-4127-92e0-57e4858252d8\" (UID: \"f477695a-21ec-4127-92e0-57e4858252d8\") "
Oct 30 22:46:05.670578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.670556    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:05.672728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.672669    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca" pod "f477695a-21ec-4127-92e0-57e4858252d8" (UID: "f477695a-21ec-4127-92e0-57e4858252d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:05.675027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.675002    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx" pod "f477695a-21ec-4127-92e0-57e4858252d8" (UID: "f477695a-21ec-4127-92e0-57e4858252d8"). InnerVolumeSpecName "kube-api-access-t6kjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:05.772417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.772381    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6kjx\" (UniqueName: \"kubernetes.io/projected/f477695a-21ec-4127-92e0-57e4858252d8-kube-api-access-t6kjx\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:05.772417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:05.772416    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f477695a-21ec-4127-92e0-57e4858252d8-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:06.407043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.407014    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"
Oct 30 22:46:06.407524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.407013    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6" event={"ID":"f477695a-21ec-4127-92e0-57e4858252d8","Type":"ContainerDied","Data":"3af641037aaf6ed30c33e7174b1feef1d5bbe5adc3c949df00561991fd7b6579"}
Oct 30 22:46:06.407524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.407136    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af641037aaf6ed30c33e7174b1feef1d5bbe5adc3c949df00561991fd7b6579"
Oct 30 22:46:06.409920 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.409893    5028 generic.go:356] "Generic (PLEG): container finished" podID="78b606d7-09d8-4a9d-97a5-6697722ead44" containerID="10e229ba72c9ebae851aa9c06e7810da0fb059a99bdb6ddab3e5386a3e6acc63" exitCode=0
Oct 30 22:46:06.410079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.410017    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4" event={"ID":"78b606d7-09d8-4a9d-97a5-6697722ead44","Type":"ContainerDied","Data":"10e229ba72c9ebae851aa9c06e7810da0fb059a99bdb6ddab3e5386a3e6acc63"}
Oct 30 22:46:06.411645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.411617    5028 generic.go:356] "Generic (PLEG): container finished" podID="d56b1a0d-e240-49b9-9eb2-2127e876a5c8" containerID="5798268c1eb85280656e6564f6967543fc3d934168776214f93458a55bbedad5" exitCode=0
Oct 30 22:46:06.411776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:06.411679    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc" event={"ID":"d56b1a0d-e240-49b9-9eb2-2127e876a5c8","Type":"ContainerDied","Data":"5798268c1eb85280656e6564f6967543fc3d934168776214f93458a55bbedad5"}
Oct 30 22:46:07.297152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.297119    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b42215-85eb-46a2-9951-944feed99b07" path="/var/lib/kubelet/pods/44b42215-85eb-46a2-9951-944feed99b07/volumes"
Oct 30 22:46:07.590772 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.590746    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:07.633947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.633908    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:07.695839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.695812    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl\" (UniqueName: \"kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl\") pod \"78b606d7-09d8-4a9d-97a5-6697722ead44\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") "
Oct 30 22:46:07.696025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.695900    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca\") pod \"78b606d7-09d8-4a9d-97a5-6697722ead44\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") "
Oct 30 22:46:07.696025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.695937    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca\") pod \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") "
Oct 30 22:46:07.696025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.695955    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca\") pod \"78b606d7-09d8-4a9d-97a5-6697722ead44\" (UID: \"78b606d7-09d8-4a9d-97a5-6697722ead44\") "
Oct 30 22:46:07.696025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.695982    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv\" (UniqueName: \"kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv\") pod \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") "
Oct 30 22:46:07.696238 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.696073    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca\") pod \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\" (UID: \"d56b1a0d-e240-49b9-9eb2-2127e876a5c8\") "
Oct 30 22:46:07.696381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.696350    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca" pod "78b606d7-09d8-4a9d-97a5-6697722ead44" (UID: "78b606d7-09d8-4a9d-97a5-6697722ead44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:07.696469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.696356    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca" pod "d56b1a0d-e240-49b9-9eb2-2127e876a5c8" (UID: "d56b1a0d-e240-49b9-9eb2-2127e876a5c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:07.696565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.696543    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:07.696634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.696571    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:07.698174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.698133    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca" pod "d56b1a0d-e240-49b9-9eb2-2127e876a5c8" (UID: "d56b1a0d-e240-49b9-9eb2-2127e876a5c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:07.698283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.698189    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca" pod "78b606d7-09d8-4a9d-97a5-6697722ead44" (UID: "78b606d7-09d8-4a9d-97a5-6697722ead44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:07.698283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.698255    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl" pod "78b606d7-09d8-4a9d-97a5-6697722ead44" (UID: "78b606d7-09d8-4a9d-97a5-6697722ead44"). InnerVolumeSpecName "kube-api-access-nkbvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:07.698362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.698287    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv" pod "d56b1a0d-e240-49b9-9eb2-2127e876a5c8" (UID: "d56b1a0d-e240-49b9-9eb2-2127e876a5c8"). InnerVolumeSpecName "kube-api-access-8lckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:07.797457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.797413    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkbvl\" (UniqueName: \"kubernetes.io/projected/78b606d7-09d8-4a9d-97a5-6697722ead44-kube-api-access-nkbvl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:07.797457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.797450    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b606d7-09d8-4a9d-97a5-6697722ead44-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:07.797457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.797464    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lckv\" (UniqueName: \"kubernetes.io/projected/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-kube-api-access-8lckv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:07.797754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:07.797476    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56b1a0d-e240-49b9-9eb2-2127e876a5c8-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:08.427874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:08.427842    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"
Oct 30 22:46:08.428102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:08.427872    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4" event={"ID":"78b606d7-09d8-4a9d-97a5-6697722ead44","Type":"ContainerDied","Data":"a384c95a115e175733ea38c961c73c049017b6b215cd88ca774ab154b43a8622"}
Oct 30 22:46:08.428102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:08.427925    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a384c95a115e175733ea38c961c73c049017b6b215cd88ca774ab154b43a8622"
Oct 30 22:46:08.429484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:08.429457    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"
Oct 30 22:46:08.429633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:08.429492    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc" event={"ID":"d56b1a0d-e240-49b9-9eb2-2127e876a5c8","Type":"ContainerDied","Data":"6006ab9b449621e8103444932f65cecc0b5f4613af4d336d7d591b28d8eff71e"}
Oct 30 22:46:08.429633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:08.429535    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6006ab9b449621e8103444932f65cecc0b5f4613af4d336d7d591b28d8eff71e"
Oct 30 22:46:09.746937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:09.746903    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:10.471314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:10.471285    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:46:10.512601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:10.512571    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:46:13.416591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:13.416551    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"]
Oct 30 22:46:13.658473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:13.658437    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:13.821946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:13.821905    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:15.282586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:15.282532    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:15.433870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:15.433825    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:17.511394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.511353    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:17.641858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.641816    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"]
Oct 30 22:46:17.642727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642706    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d56b1a0d-e240-49b9-9eb2-2127e876a5c8" containerName="revoke-keys"
Oct 30 22:46:17.642727 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642726    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56b1a0d-e240-49b9-9eb2-2127e876a5c8" containerName="revoke-keys"
Oct 30 22:46:17.642933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642769    5028 cpu_manager.go:414] "containerMap: removing container" podUID="78b606d7-09d8-4a9d-97a5-6697722ead44" containerName="revoke-keys"
Oct 30 22:46:17.642933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642777    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b606d7-09d8-4a9d-97a5-6697722ead44" containerName="revoke-keys"
Oct 30 22:46:17.642933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642792    5028 cpu_manager.go:414] "containerMap: removing container" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main"
Oct 30 22:46:17.642933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642799    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main"
Oct 30 22:46:17.642933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642818    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f477695a-21ec-4127-92e0-57e4858252d8" containerName="revoke-keys"
Oct 30 22:46:17.642933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642825    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f477695a-21ec-4127-92e0-57e4858252d8" containerName="revoke-keys"
Oct 30 22:46:17.643284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642959    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f477695a-21ec-4127-92e0-57e4858252d8" containerName="revoke-keys"
Oct 30 22:46:17.643284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642976    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56b1a0d-e240-49b9-9eb2-2127e876a5c8" containerName="revoke-keys"
Oct 30 22:46:17.643284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.642993    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b606d7-09d8-4a9d-97a5-6697722ead44" containerName="revoke-keys"
Oct 30 22:46:17.643284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.643003    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b42215-85eb-46a2-9951-944feed99b07" containerName="main"
Oct 30 22:46:17.648262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.648235    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.700319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.700287    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"]
Oct 30 22:46:17.723605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.723570    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.723816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.723738    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdqr\" (UniqueName: \"kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.723816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.723799    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.740997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.740967    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:17.741145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.741081    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:46:17.746190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.746174    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:17.746425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.746410    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:17.764543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.764480    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:17.825138 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.825103    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdqr\" (UniqueName: \"kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.825315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.825165    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.825315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.825287    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.825898 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.825865    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.827668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.827639    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.885857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.885824    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdqr\" (UniqueName: \"kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr\") pod \"maas-api-revoke-sub-f3fd866c-zv2p7\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:17.962391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:17.962354    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:18.011728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.011338    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"]
Oct 30 22:46:18.015434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.015359    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6b55f-wq8j6"]
Oct 30 22:46:18.343864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.343833    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"]
Oct 30 22:46:18.344490 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:46:18.344461    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f14f08_635d_4bd0_9c5b_32a5f62fc207.slice/crio-a4777289c58e3cb1f5ce66b5286896a52772b11da5621a175de7f20720724fbb WatchSource:0}: Error finding container a4777289c58e3cb1f5ce66b5286896a52772b11da5621a175de7f20720724fbb: Status 404 returned error can't find the container with id a4777289c58e3cb1f5ce66b5286896a52772b11da5621a175de7f20720724fbb
Oct 30 22:46:18.417470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.417428    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerName="payload-pre-processing" containerID="cri-o://32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427" gracePeriod=25
Oct 30 22:46:18.452085 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.452054    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:18.539039 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.539007    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7" event={"ID":"b1f14f08-635d-4bd0-9c5b-32a5f62fc207","Type":"ContainerStarted","Data":"a4777289c58e3cb1f5ce66b5286896a52772b11da5621a175de7f20720724fbb"}
Oct 30 22:46:18.858752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.858718    5028 scope.go:122] "RemoveContainer" containerID="8227324ed15951408b501e879dddc7f4566b204c08882ebfb7bfe87dfa0f45f5"
Oct 30 22:46:18.870152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.870131    5028 scope.go:122] "RemoveContainer" containerID="2ad1bc5ae897e399446bba64144bd2135a06f30b898652653ce72b3b187c9bde"
Oct 30 22:46:18.879482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.879454    5028 scope.go:122] "RemoveContainer" containerID="7b5eba30fca24c9d248cca49cf99bb5eb34756451966df418188395cde61e961"
Oct 30 22:46:18.893372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.893348    5028 scope.go:122] "RemoveContainer" containerID="0654f497886b57ed57a0fb0f4ad1e2f73f179b585eb335e9b8c8db3f16c110b3"
Oct 30 22:46:18.902778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:18.902758    5028 scope.go:122] "RemoveContainer" containerID="622d1ef67a4afd7b4d4aa7ea9b238ee81d3db9fb50ca72e6897488d9c0f00a2f"
Oct 30 22:46:19.292937 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:19.292901    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f477695a-21ec-4127-92e0-57e4858252d8" path="/var/lib/kubelet/pods/f477695a-21ec-4127-92e0-57e4858252d8/volumes"
Oct 30 22:46:19.549271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:19.549174    5028 generic.go:356] "Generic (PLEG): container finished" podID="b1f14f08-635d-4bd0-9c5b-32a5f62fc207" containerID="1f0c1866acec303155a81d25184a84dfd7375f6e7bad36c043962d897f129b22" exitCode=0
Oct 30 22:46:19.549271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:19.549261    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7" event={"ID":"b1f14f08-635d-4bd0-9c5b-32a5f62fc207","Type":"ContainerDied","Data":"1f0c1866acec303155a81d25184a84dfd7375f6e7bad36c043962d897f129b22"}
Oct 30 22:46:20.276866 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.276828    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:20.729645 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.729622    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:20.865245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.865152    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr\" (UniqueName: \"kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr\") pod \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") "
Oct 30 22:46:20.865245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.865211    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca\") pod \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") "
Oct 30 22:46:20.865530 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.865313    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca\") pod \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\" (UID: \"b1f14f08-635d-4bd0-9c5b-32a5f62fc207\") "
Oct 30 22:46:20.868909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.868880    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca" pod "b1f14f08-635d-4bd0-9c5b-32a5f62fc207" (UID: "b1f14f08-635d-4bd0-9c5b-32a5f62fc207"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:20.871096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.871067    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca" pod "b1f14f08-635d-4bd0-9c5b-32a5f62fc207" (UID: "b1f14f08-635d-4bd0-9c5b-32a5f62fc207"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:20.871366 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.871346    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr" pod "b1f14f08-635d-4bd0-9c5b-32a5f62fc207" (UID: "b1f14f08-635d-4bd0-9c5b-32a5f62fc207"). InnerVolumeSpecName "kube-api-access-jwdqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:20.966748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.966697    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdqr\" (UniqueName: \"kubernetes.io/projected/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-kube-api-access-jwdqr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:20.966748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.966730    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:20.966748 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:20.966739    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f14f08-635d-4bd0-9c5b-32a5f62fc207-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:21.566316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:21.566216    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"
Oct 30 22:46:21.566497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:21.566299    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7" event={"ID":"b1f14f08-635d-4bd0-9c5b-32a5f62fc207","Type":"ContainerDied","Data":"a4777289c58e3cb1f5ce66b5286896a52772b11da5621a175de7f20720724fbb"}
Oct 30 22:46:21.566497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:21.566351    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4777289c58e3cb1f5ce66b5286896a52772b11da5621a175de7f20720724fbb"
Oct 30 22:46:24.989111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:24.989074    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:25.153013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:25.152976    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:25.312960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:25.312868    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:25.417648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:25.417620    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"]
Oct 30 22:46:25.417873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:25.417846    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy" containerID="cri-o://84e955f10d58ee87344fb5859212ca863d792ee746d30efb33d07e4dcd8210dd" gracePeriod=30
Oct 30 22:46:25.459015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:25.458983    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:26.288969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.288936    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"]
Oct 30 22:46:26.289555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.289538    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b1f14f08-635d-4bd0-9c5b-32a5f62fc207" containerName="revoke-keys"
Oct 30 22:46:26.289618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.289557    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f14f08-635d-4bd0-9c5b-32a5f62fc207" containerName="revoke-keys"
Oct 30 22:46:26.289671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.289661    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f14f08-635d-4bd0-9c5b-32a5f62fc207" containerName="revoke-keys"
Oct 30 22:46:26.293540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.293522    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.348011 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.347983    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"]
Oct 30 22:46:26.390075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.390041    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:26.390253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.390123    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:26.390253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.390194    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:46:26.390253 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.390203    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:26.429670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.429637    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.429859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.429678    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.429859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.429829    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6sw\" (UniqueName: \"kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.530611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.530581    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.530836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.530620    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.530836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.530667    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6sw\" (UniqueName: \"kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.532184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.532155    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.533556 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.533530    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.590045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.589958    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6sw\" (UniqueName: \"kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw\") pod \"maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.608031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.607998    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:26.985570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:26.985538    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"]
Oct 30 22:46:26.986740 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:46:26.986710    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97973dbd_f74d_43c7_a8ae_a8d1edf62c06.slice/crio-1e7b8405075189783f61b949a406e180569d8b98c6eeb19ccde1b02509627b62 WatchSource:0}: Error finding container 1e7b8405075189783f61b949a406e180569d8b98c6eeb19ccde1b02509627b62: Status 404 returned error can't find the container with id 1e7b8405075189783f61b949a406e180569d8b98c6eeb19ccde1b02509627b62
Oct 30 22:46:27.462933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:27.462901    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:27.616057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:27.615952    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g" event={"ID":"97973dbd-f74d-43c7-a8ae-a8d1edf62c06","Type":"ContainerStarted","Data":"800a77ddd894fc66f358bdcc5e4d0411431e7fdafad8f8c31e1a90b301bd8f9d"}
Oct 30 22:46:27.616057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:27.616022    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g" event={"ID":"97973dbd-f74d-43c7-a8ae-a8d1edf62c06","Type":"ContainerStarted","Data":"1e7b8405075189783f61b949a406e180569d8b98c6eeb19ccde1b02509627b62"}
Oct 30 22:46:27.626041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:27.626002    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:27.732321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:27.732260    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g" podStartSLOduration=1.73224629 podStartE2EDuration="1.73224629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:27.729405254 +0000 UTC m=+2718.578918227" watchObservedRunningTime="2026-10-30 22:46:27.73224629 +0000 UTC m=+2718.581759263"
Oct 30 22:46:28.126762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:28.126728    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:28.436833 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:46:28.436744    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:46:28.625383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:28.625339    5028 generic.go:356] "Generic (PLEG): container finished" podID="97973dbd-f74d-43c7-a8ae-a8d1edf62c06" containerID="800a77ddd894fc66f358bdcc5e4d0411431e7fdafad8f8c31e1a90b301bd8f9d" exitCode=0
Oct 30 22:46:28.627369 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:28.625440    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g" event={"ID":"97973dbd-f74d-43c7-a8ae-a8d1edf62c06","Type":"ContainerDied","Data":"800a77ddd894fc66f358bdcc5e4d0411431e7fdafad8f8c31e1a90b301bd8f9d"}
Oct 30 22:46:29.754796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.754763    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:29.807995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.807965    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:29.974080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.974048    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw\" (UniqueName: \"kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw\") pod \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") "
Oct 30 22:46:29.974305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.974102    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca\") pod \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") "
Oct 30 22:46:29.974305 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.974159    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca\") pod \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\" (UID: \"97973dbd-f74d-43c7-a8ae-a8d1edf62c06\") "
Oct 30 22:46:29.974636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.974601    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca" pod "97973dbd-f74d-43c7-a8ae-a8d1edf62c06" (UID: "97973dbd-f74d-43c7-a8ae-a8d1edf62c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:29.976394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.976370    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw" pod "97973dbd-f74d-43c7-a8ae-a8d1edf62c06" (UID: "97973dbd-f74d-43c7-a8ae-a8d1edf62c06"). InnerVolumeSpecName "kube-api-access-pp6sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:29.976501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:29.976431    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca" pod "97973dbd-f74d-43c7-a8ae-a8d1edf62c06" (UID: "97973dbd-f74d-43c7-a8ae-a8d1edf62c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:30.075376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.075335    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp6sw\" (UniqueName: \"kubernetes.io/projected/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-kube-api-access-pp6sw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.075376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.075372    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.075603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.075388    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97973dbd-f74d-43c7-a8ae-a8d1edf62c06-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.643993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.643959    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"
Oct 30 22:46:30.644199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.643983    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g" event={"ID":"97973dbd-f74d-43c7-a8ae-a8d1edf62c06","Type":"ContainerDied","Data":"1e7b8405075189783f61b949a406e180569d8b98c6eeb19ccde1b02509627b62"}
Oct 30 22:46:30.644199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.644054    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7b8405075189783f61b949a406e180569d8b98c6eeb19ccde1b02509627b62"
Oct 30 22:46:30.646362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.646337    5028 generic.go:356] "Generic (PLEG): container finished" podID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerID="84e955f10d58ee87344fb5859212ca863d792ee746d30efb33d07e4dcd8210dd" exitCode=0
Oct 30 22:46:30.646525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.646413    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" event={"ID":"d91a055c-7859-4bf5-8e64-25d29b12a7af","Type":"ContainerDied","Data":"84e955f10d58ee87344fb5859212ca863d792ee746d30efb33d07e4dcd8210dd"}
Oct 30 22:46:30.646525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.646477    5028 scope.go:122] "RemoveContainer" containerID="6cf6c97880eb6322f3dc2dde625faca51dbf9e062438f8630355dd97b306f46c"
Oct 30 22:46:30.667383 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.667362    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:46:30.682775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.682707    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.682775 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.682770    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.682953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.682817    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.682953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.682851    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.682953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.682883    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.682953 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.682905    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.683212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.683186    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:30.683271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.683215    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:30.683367 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.683353    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91a055c-7859-4bf5-8e64-25d29b12a7af-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.683435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.683370    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.683435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.683369    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:30.685228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.685205    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:30.685228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.685218    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "kube-api-access-csb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:30.685399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.685274    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:46:30.783966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.783933    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.784391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784009    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.784391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784070    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy\") pod \"d91a055c-7859-4bf5-8e64-25d29b12a7af\" (UID: \"d91a055c-7859-4bf5-8e64-25d29b12a7af\") "
Oct 30 22:46:30.784391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784262    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:30.784391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784274    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:30.784545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784507    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.784545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784528    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csb72\" (UniqueName: \"kubernetes.io/projected/d91a055c-7859-4bf5-8e64-25d29b12a7af-kube-api-access-csb72\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.784607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784542    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.784607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784556    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.784607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784570    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.784607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.784585    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:30.786106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.786087    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy" pod "d91a055c-7859-4bf5-8e64-25d29b12a7af" (UID: "d91a055c-7859-4bf5-8e64-25d29b12a7af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:30.886023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:30.885989    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91a055c-7859-4bf5-8e64-25d29b12a7af-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:31.655162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:31.655084    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"
Oct 30 22:46:31.655162 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:31.655109    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn" event={"ID":"d91a055c-7859-4bf5-8e64-25d29b12a7af","Type":"ContainerDied","Data":"478cba9c26daf832e88457ddd60dd1cafae24e832072418a0e67b479e558d439"}
Oct 30 22:46:31.655376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:31.655164    5028 scope.go:122] "RemoveContainer" containerID="84e955f10d58ee87344fb5859212ca863d792ee746d30efb33d07e4dcd8210dd"
Oct 30 22:46:31.815683 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:31.815649    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"]
Oct 30 22:46:31.862984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:31.862942    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d6b55f-openshift-default-68c88686bc-bf4jn"]
Oct 30 22:46:32.335790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:32.335756    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:32.510985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:32.510948    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:33.116328 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:33.116293    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:33.278728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:33.278660    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:33.291656 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:33.291626    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" path="/var/lib/kubelet/pods/d91a055c-7859-4bf5-8e64-25d29b12a7af/volumes"
Oct 30 22:46:34.125246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125207    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"]
Oct 30 22:46:34.125839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125822    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy"
Oct 30 22:46:34.125897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125840    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy"
Oct 30 22:46:34.125897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125857    5028 cpu_manager.go:414] "containerMap: removing container" podUID="97973dbd-f74d-43c7-a8ae-a8d1edf62c06" containerName="revoke-keys"
Oct 30 22:46:34.125897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125862    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97973dbd-f74d-43c7-a8ae-a8d1edf62c06" containerName="revoke-keys"
Oct 30 22:46:34.125897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125872    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy"
Oct 30 22:46:34.125897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125878    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy"
Oct 30 22:46:34.126068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125982    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy"
Oct 30 22:46:34.126068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125989    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="97973dbd-f74d-43c7-a8ae-a8d1edf62c06" containerName="revoke-keys"
Oct 30 22:46:34.126068 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.125997    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91a055c-7859-4bf5-8e64-25d29b12a7af" containerName="istio-proxy"
Oct 30 22:46:34.131240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.131219    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.187348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.187317    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"]
Oct 30 22:46:34.222292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.222259    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cb4\" (UniqueName: \"kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.222466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.222438    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.222549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.222488    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.228662 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.228634    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:34.228835 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.228704    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:46:34.231903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.231879    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:34.231903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.231891    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:34.306210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.306180    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:34.324733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.324537    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.324733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.324592    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.325763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.324721    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cb4\" (UniqueName: \"kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.328730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.326090    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.331043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.331010    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.386474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.386392    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cb4\" (UniqueName: \"kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4\") pod \"maas-api-revoke-sub-e3e5e902-6lldw\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.448845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.448812    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:34.830651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:34.830620    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"]
Oct 30 22:46:34.831236 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:46:34.831211    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0790b25_cf84_4869_86a8_398ae57794b3.slice/crio-aa64667d9dd40f0515e65256be4145a9b7852fcf210debab74c1c799dc7ceb3e WatchSource:0}: Error finding container aa64667d9dd40f0515e65256be4145a9b7852fcf210debab74c1c799dc7ceb3e: Status 404 returned error can't find the container with id aa64667d9dd40f0515e65256be4145a9b7852fcf210debab74c1c799dc7ceb3e
Oct 30 22:46:35.312194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:35.312156    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:35.689618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:35.689583    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw" event={"ID":"a0790b25-cf84-4869-86a8-398ae57794b3","Type":"ContainerStarted","Data":"e05b87115fa899dc8bd4f285f7c10a999324515ee5ece0d74e1bd61ea0f97fe9"}
Oct 30 22:46:35.689830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:35.689643    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw" event={"ID":"a0790b25-cf84-4869-86a8-398ae57794b3","Type":"ContainerStarted","Data":"aa64667d9dd40f0515e65256be4145a9b7852fcf210debab74c1c799dc7ceb3e"}
Oct 30 22:46:35.802807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:35.802751    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw" podStartSLOduration=1.802736757 podStartE2EDuration="1.802736757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:35.800895894 +0000 UTC m=+2726.650408868" watchObservedRunningTime="2026-10-30 22:46:35.802736757 +0000 UTC m=+2726.652249762"
Oct 30 22:46:36.698449 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:36.698418    5028 generic.go:356] "Generic (PLEG): container finished" podID="a0790b25-cf84-4869-86a8-398ae57794b3" containerID="e05b87115fa899dc8bd4f285f7c10a999324515ee5ece0d74e1bd61ea0f97fe9" exitCode=0
Oct 30 22:46:36.698880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:36.698489    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw" event={"ID":"a0790b25-cf84-4869-86a8-398ae57794b3","Type":"ContainerDied","Data":"e05b87115fa899dc8bd4f285f7c10a999324515ee5ece0d74e1bd61ea0f97fe9"}
Oct 30 22:46:37.231089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.231047    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"]
Oct 30 22:46:37.334509 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.334473    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"]
Oct 30 22:46:37.570926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.570846    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:37.722840 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.722797    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:37.868300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.868275    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:37.968929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.968897    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca\") pod \"a0790b25-cf84-4869-86a8-398ae57794b3\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") "
Oct 30 22:46:37.969149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.969088    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4\" (UniqueName: \"kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4\") pod \"a0790b25-cf84-4869-86a8-398ae57794b3\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") "
Oct 30 22:46:37.969149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.969120    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca\") pod \"a0790b25-cf84-4869-86a8-398ae57794b3\" (UID: \"a0790b25-cf84-4869-86a8-398ae57794b3\") "
Oct 30 22:46:37.969575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.969546    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca" pod "a0790b25-cf84-4869-86a8-398ae57794b3" (UID: "a0790b25-cf84-4869-86a8-398ae57794b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:37.971123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.971099    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca" pod "a0790b25-cf84-4869-86a8-398ae57794b3" (UID: "a0790b25-cf84-4869-86a8-398ae57794b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:37.971203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:37.971166    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4" pod "a0790b25-cf84-4869-86a8-398ae57794b3" (UID: "a0790b25-cf84-4869-86a8-398ae57794b3"). InnerVolumeSpecName "kube-api-access-b9cb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:38.070474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.070439    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:38.070474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.070467    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9cb4\" (UniqueName: \"kubernetes.io/projected/a0790b25-cf84-4869-86a8-398ae57794b3-kube-api-access-b9cb4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:38.070474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.070477    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0790b25-cf84-4869-86a8-398ae57794b3-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:38.710917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.710881    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:46:38.716430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.716402    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"
Oct 30 22:46:38.716627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.716429    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw" event={"ID":"a0790b25-cf84-4869-86a8-398ae57794b3","Type":"ContainerDied","Data":"aa64667d9dd40f0515e65256be4145a9b7852fcf210debab74c1c799dc7ceb3e"}
Oct 30 22:46:38.716627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.716478    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa64667d9dd40f0515e65256be4145a9b7852fcf210debab74c1c799dc7ceb3e"
Oct 30 22:46:38.718295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.718264    5028 generic.go:356] "Generic (PLEG): container finished" podID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerID="32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427" exitCode=0
Oct 30 22:46:38.718454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.718298    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" event={"ID":"adcf5007-2264-4a1d-b579-9e6fce8f9f75","Type":"ContainerDied","Data":"32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427"}
Oct 30 22:46:38.718454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.718310    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"
Oct 30 22:46:38.718454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.718370    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs" event={"ID":"adcf5007-2264-4a1d-b579-9e6fce8f9f75","Type":"ContainerDied","Data":"e6914e8b56826166c6f0d39588a2b39d883635174a9748ec9ab4481a0e7cf018"}
Oct 30 22:46:38.718454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.718398    5028 scope.go:122] "RemoveContainer" containerID="32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427"
Oct 30 22:46:38.731075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.731052    5028 scope.go:122] "RemoveContainer" containerID="32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427"
Oct 30 22:46:38.731467 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:46:38.731423    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427\": container with ID starting with 32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427 not found: ID does not exist" containerID="32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427"
Oct 30 22:46:38.731536 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.731462    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427"} err="failed to get container status \"32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427\": rpc error: code = NotFound desc = could not find container \"32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427\": container with ID starting with 32e7b4a04aaa8d2297f4eeabe76a8efd3e4189f48664df8c09c5aea1026fd427 not found: ID does not exist"
Oct 30 22:46:38.775278 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.775233    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:38.783401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.783369    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume\") pod \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") "
Oct 30 22:46:38.783585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.783408    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52\" (UniqueName: \"kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52\") pod \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\" (UID: \"adcf5007-2264-4a1d-b579-9e6fce8f9f75\") "
Oct 30 22:46:38.783841 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.783764    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume" pod "adcf5007-2264-4a1d-b579-9e6fce8f9f75" (UID: "adcf5007-2264-4a1d-b579-9e6fce8f9f75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:38.784098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.784078    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcf5007-2264-4a1d-b579-9e6fce8f9f75-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:38.785722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.785676    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52" pod "adcf5007-2264-4a1d-b579-9e6fce8f9f75" (UID: "adcf5007-2264-4a1d-b579-9e6fce8f9f75"). InnerVolumeSpecName "kube-api-access-h6h52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:38.885472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:38.885439    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6h52\" (UniqueName: \"kubernetes.io/projected/adcf5007-2264-4a1d-b579-9e6fce8f9f75-kube-api-access-h6h52\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:39.138800 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:39.138751    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"]
Oct 30 22:46:39.144501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:39.144458    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6b55f-599cf486f5-kbgrs"]
Oct 30 22:46:39.290658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:39.290569    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" path="/var/lib/kubelet/pods/adcf5007-2264-4a1d-b579-9e6fce8f9f75/volumes"
Oct 30 22:46:42.231862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:42.231801    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerName="payload-processing" containerID="cri-o://430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd" gracePeriod=25
Oct 30 22:46:42.335255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:42.335176    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerName="payload-pre-processing" containerID="cri-o://1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283" gracePeriod=25
Oct 30 22:46:43.456904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:43.456868    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:47.447306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:47.447255    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:48.835868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:48.835823    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.013909 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.013866    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.183961 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.183849    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.310401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.310367    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"]
Oct 30 22:46:49.357419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.357380    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23f3d4-2rgfs"]
Oct 30 22:46:49.781880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.781799    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:49.951753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.951714    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:49.952222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:49.951765    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:51.049172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:51.049121    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"]
Oct 30 22:46:51.095888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:51.095846    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0124df-65q46"]
Oct 30 22:46:51.292057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:51.292015    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543a5569-ca67-438c-befd-a693fd7c7a08" path="/var/lib/kubelet/pods/543a5569-ca67-438c-befd-a693fd7c7a08/volumes"
Oct 30 22:46:51.292485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:51.292471    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ab0e5e-ba97-40d6-8455-271bae592cf4" path="/var/lib/kubelet/pods/e4ab0e5e-ba97-40d6-8455-271bae592cf4/volumes"
Oct 30 22:46:53.944939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:53.944902    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:56.173700 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:56.173652    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:56.330222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:56.330193    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:56.513488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:56.513443    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"]
Oct 30 22:46:56.513712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:56.513669    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" containerID="cri-o://de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f" gracePeriod=30
Oct 30 22:46:56.543896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:56.543866    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:58.436416 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:46:58.436370    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:46:58.533956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:58.533918    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:59.951567 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:59.951524    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:59.952009 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:46:59.951529    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:01.598425 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:01.598387    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd86ee34_1f3f_4bf6_b93c_ccb204fdf1a9.slice/crio-conmon-de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:47:01.627286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.627253    5028 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:47:01.627444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.627304    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 22:47:01.798237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.798211    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:47:01.891836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.891803    5028 generic.go:356] "Generic (PLEG): container finished" podID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerID="de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f" exitCode=0
Oct 30 22:47:01.892040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.891851    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"
Oct 30 22:47:01.892040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.891886    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" event={"ID":"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9","Type":"ContainerDied","Data":"de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f"}
Oct 30 22:47:01.892040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.891947    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m" event={"ID":"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9","Type":"ContainerDied","Data":"3b54ae801323717b953164a774d9112f7dd1cddf1a27c49ce8a0a363c9dbdb52"}
Oct 30 22:47:01.892040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.891972    5028 scope.go:122] "RemoveContainer" containerID="de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f"
Oct 30 22:47:01.906905 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.906879    5028 scope.go:122] "RemoveContainer" containerID="a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"
Oct 30 22:47:01.916669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.916648    5028 scope.go:122] "RemoveContainer" containerID="de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f"
Oct 30 22:47:01.916996 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:01.916975    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f\": container with ID starting with de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f not found: ID does not exist" containerID="de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f"
Oct 30 22:47:01.917088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.917005    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f"} err="failed to get container status \"de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f\": rpc error: code = NotFound desc = could not find container \"de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f\": container with ID starting with de3ffe553bf0f4bb68fb71b164edc79937a6bf6dac4d7d7f5f4d92635d3da25f not found: ID does not exist"
Oct 30 22:47:01.917088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.917026    5028 scope.go:122] "RemoveContainer" containerID="a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"
Oct 30 22:47:01.917348 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:01.917324    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab\": container with ID starting with a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab not found: ID does not exist" containerID="a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"
Oct 30 22:47:01.917413 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.917358    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab"} err="failed to get container status \"a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab\": rpc error: code = NotFound desc = could not find container \"a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab\": container with ID starting with a6e8735462c73c4f2d82cab0aa840f479b8b1485561fc261111d9af6bb6630ab not found: ID does not exist"
Oct 30 22:47:01.966082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966045    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966093    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966129    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966266    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966345    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966369    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:01.966570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966399    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966482    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:01.966570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966512    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966570 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966548    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966614    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:01.966842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966628    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy\") pod \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\" (UID: \"dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9\") "
Oct 30 22:47:01.966987 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.966841    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:01.967168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.967140    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:01.967257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.967149    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:01.967257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.967223    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:01.967257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.967241    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:01.967395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.967265    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:01.969728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.969645    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "kube-api-access-nmcvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:01.970010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.969987    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:01.970482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.970458    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:01.970829 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:01.970807    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token" pod "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" (UID: "dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:02.068605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.068516    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcvk\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-kube-api-access-nmcvk\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.068605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.068549    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.068605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.068559    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.068605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.068569    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.068605 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.068579    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.312559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.312526    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"]
Oct 30 22:47:02.317426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.317396    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-p956m"]
Oct 30 22:47:02.500497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.499553    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:02.634813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.630822    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:47:02.680585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.680560    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:47:02.687849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.687822    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:02.784214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.784175    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq\" (UniqueName: \"kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq\") pod \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") "
Oct 30 22:47:02.784469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.784283    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume\") pod \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\" (UID: \"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1\") "
Oct 30 22:47:02.784469 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.784325    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume\") pod \"4bde795e-1061-46dc-bdd7-71f5a27dede1\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") "
Oct 30 22:47:02.792324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.788651    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj\" (UniqueName: \"kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj\") pod \"4bde795e-1061-46dc-bdd7-71f5a27dede1\" (UID: \"4bde795e-1061-46dc-bdd7-71f5a27dede1\") "
Oct 30 22:47:02.792324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.791245    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume" pod "d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" (UID: "d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:02.792324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.791881    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.792869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.792834    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume" pod "4bde795e-1061-46dc-bdd7-71f5a27dede1" (UID: "4bde795e-1061-46dc-bdd7-71f5a27dede1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:02.795792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.795756    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq" pod "d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" (UID: "d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1"). InnerVolumeSpecName "kube-api-access-7h2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:02.796308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.796261    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj" pod "4bde795e-1061-46dc-bdd7-71f5a27dede1" (UID: "4bde795e-1061-46dc-bdd7-71f5a27dede1"). InnerVolumeSpecName "kube-api-access-2r2gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:02.893614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.893522    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bde795e-1061-46dc-bdd7-71f5a27dede1-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.893614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.893562    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2gj\" (UniqueName: \"kubernetes.io/projected/4bde795e-1061-46dc-bdd7-71f5a27dede1-kube-api-access-2r2gj\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.893614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.893578    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2fq\" (UniqueName: \"kubernetes.io/projected/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1-kube-api-access-7h2fq\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:02.914059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.912112    5028 generic.go:356] "Generic (PLEG): container finished" podID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerID="1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283" exitCode=0
Oct 30 22:47:02.914059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.912246    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" event={"ID":"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1","Type":"ContainerDied","Data":"1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283"}
Oct 30 22:47:02.914059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.912345    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm" event={"ID":"d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1","Type":"ContainerDied","Data":"c25bad7f0990b0c29f1ae3d9b5379f1e35bcf15ce5c2a1edb618a8b5bb29128f"}
Oct 30 22:47:02.914059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.912390    5028 scope.go:122] "RemoveContainer" containerID="1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283"
Oct 30 22:47:02.914059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.912587    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"
Oct 30 22:47:02.931915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.931875    5028 generic.go:356] "Generic (PLEG): container finished" podID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerID="430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd" exitCode=0
Oct 30 22:47:02.932179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.931933    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" event={"ID":"4bde795e-1061-46dc-bdd7-71f5a27dede1","Type":"ContainerDied","Data":"430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd"}
Oct 30 22:47:02.932179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.932076    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"
Oct 30 22:47:02.932179 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.932129    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw" event={"ID":"4bde795e-1061-46dc-bdd7-71f5a27dede1","Type":"ContainerDied","Data":"74624416e9c7699c20cbf57c8c9bc132ae51a448dacd0bbe1e366398c269209b"}
Oct 30 22:47:02.951824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.951800    5028 scope.go:122] "RemoveContainer" containerID="1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283"
Oct 30 22:47:02.952160 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:02.952137    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283\": container with ID starting with 1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283 not found: ID does not exist" containerID="1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283"
Oct 30 22:47:02.952264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.952171    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283"} err="failed to get container status \"1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283\": rpc error: code = NotFound desc = could not find container \"1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283\": container with ID starting with 1954a844f377bfe0ee52ed96ac752350576c444b73ed6391cb48f5d3a72f9283 not found: ID does not exist"
Oct 30 22:47:02.952264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.952196    5028 scope.go:122] "RemoveContainer" containerID="430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd"
Oct 30 22:47:02.961667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.961638    5028 scope.go:122] "RemoveContainer" containerID="430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd"
Oct 30 22:47:02.961964 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:02.961943    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd\": container with ID starting with 430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd not found: ID does not exist" containerID="430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd"
Oct 30 22:47:02.962028 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:02.961974    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd"} err="failed to get container status \"430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd\": rpc error: code = NotFound desc = could not find container \"430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd\": container with ID starting with 430c7b94fb12f1eb7ee86796e15662cde97386c42379d5050ae0cfdc641986cd not found: ID does not exist"
Oct 30 22:47:03.014785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.014753    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"]
Oct 30 22:47:03.015293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015280    5028 cpu_manager.go:414] "containerMap: removing container" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerName="payload-pre-processing"
Oct 30 22:47:03.015338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015294    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerName="payload-pre-processing"
Oct 30 22:47:03.015338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015316    5028 cpu_manager.go:414] "containerMap: removing container" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy"
Oct 30 22:47:03.015338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015321    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy"
Oct 30 22:47:03.015338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015333    5028 cpu_manager.go:414] "containerMap: removing container" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy"
Oct 30 22:47:03.015338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015337    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015351    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a0790b25-cf84-4869-86a8-398ae57794b3" containerName="revoke-keys"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015357    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0790b25-cf84-4869-86a8-398ae57794b3" containerName="revoke-keys"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015370    5028 cpu_manager.go:414] "containerMap: removing container" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerName="payload-pre-processing"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015375    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerName="payload-pre-processing"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015385    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerName="payload-processing"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015390    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerName="payload-processing"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015460    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" containerName="payload-processing"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015470    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcf5007-2264-4a1d-b579-9e6fce8f9f75" containerName="payload-pre-processing"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015480    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015488    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" containerName="istio-proxy"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015496    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0790b25-cf84-4869-86a8-398ae57794b3" containerName="revoke-keys"
Oct 30 22:47:03.015504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.015503    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" containerName="payload-pre-processing"
Oct 30 22:47:03.020586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.020565    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.072107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.072078    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:03.072282 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.072178    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:03.075516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.075491    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:03.075985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.075822    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:47:03.076430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.076412    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"]
Oct 30 22:47:03.081402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.080886    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"]
Oct 30 22:47:03.122096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.122060    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d585d2-67f4bdbdf9-9vfbm"]
Oct 30 22:47:03.197076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.196981    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.197076 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.197029    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmbz\" (UniqueName: \"kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.197275 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.197218    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.226846 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.226806    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"]
Oct 30 22:47:03.232286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.232256    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d585d2-747cfb8979-7tgkw"]
Oct 30 22:47:03.290669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.290638    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bde795e-1061-46dc-bdd7-71f5a27dede1" path="/var/lib/kubelet/pods/4bde795e-1061-46dc-bdd7-71f5a27dede1/volumes"
Oct 30 22:47:03.291104 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.291091    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1" path="/var/lib/kubelet/pods/d2d1bfc4-14e7-4869-9c41-4b9c17dcb4a1/volumes"
Oct 30 22:47:03.291517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.291505    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9" path="/var/lib/kubelet/pods/dd86ee34-1f3f-4bf6-b93c-ccb204fdf1a9/volumes"
Oct 30 22:47:03.298742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.298717    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmbz\" (UniqueName: \"kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.298836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.298807    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.298895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.298856    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.299478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.299454    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.301116 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.301100    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.378114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.378080    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmbz\" (UniqueName: \"kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz\") pod \"maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:03.634665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:03.634628    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:04.013150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.013119    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"]
Oct 30 22:47:04.013150 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:47:04.013138    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc690e8a2_7d89_4924_ba1c_3854eaffd19a.slice/crio-8bd08011e538d828b6f2d2ed7b2505886facd67ac766dbc1b299b68da3abf06d WatchSource:0}: Error finding container 8bd08011e538d828b6f2d2ed7b2505886facd67ac766dbc1b299b68da3abf06d: Status 404 returned error can't find the container with id 8bd08011e538d828b6f2d2ed7b2505886facd67ac766dbc1b299b68da3abf06d
Oct 30 22:47:04.031102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.031071    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:04.291602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.291521    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:04.488141 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.488096    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"]
Oct 30 22:47:04.491292 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.491270    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d585d2-fqj7g"]
Oct 30 22:47:04.493520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.493496    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:04.960860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.960818    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4" event={"ID":"c690e8a2-7d89-4924-ba1c-3854eaffd19a","Type":"ContainerStarted","Data":"09018c129d8099b5edde0d821f2fc65942d7f4cd46f527ab6a0448c845294ce3"}
Oct 30 22:47:04.961095 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.960882    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4" event={"ID":"c690e8a2-7d89-4924-ba1c-3854eaffd19a","Type":"ContainerStarted","Data":"8bd08011e538d828b6f2d2ed7b2505886facd67ac766dbc1b299b68da3abf06d"}
Oct 30 22:47:04.975422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:04.975388    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:05.074286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:05.074231    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4" podStartSLOduration=3.074216135 podStartE2EDuration="3.074216135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:05.073279408 +0000 UTC m=+2755.922792381" watchObservedRunningTime="2026-10-30 22:47:05.074216135 +0000 UTC m=+2755.923729107"
Oct 30 22:47:05.291455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:05.291379    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97973dbd-f74d-43c7-a8ae-a8d1edf62c06" path="/var/lib/kubelet/pods/97973dbd-f74d-43c7-a8ae-a8d1edf62c06/volumes"
Oct 30 22:47:05.972817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:05.972783    5028 generic.go:356] "Generic (PLEG): container finished" podID="c690e8a2-7d89-4924-ba1c-3854eaffd19a" containerID="09018c129d8099b5edde0d821f2fc65942d7f4cd46f527ab6a0448c845294ce3" exitCode=0
Oct 30 22:47:05.973008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:05.972869    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4" event={"ID":"c690e8a2-7d89-4924-ba1c-3854eaffd19a","Type":"ContainerDied","Data":"09018c129d8099b5edde0d821f2fc65942d7f4cd46f527ab6a0448c845294ce3"}
Oct 30 22:47:07.139211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.139182    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:07.149365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.149335    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca\") pod \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") "
Oct 30 22:47:07.149534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.149389    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca\") pod \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") "
Oct 30 22:47:07.149678 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.149658    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz\" (UniqueName: \"kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz\") pod \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\" (UID: \"c690e8a2-7d89-4924-ba1c-3854eaffd19a\") "
Oct 30 22:47:07.149786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.149676    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca" pod "c690e8a2-7d89-4924-ba1c-3854eaffd19a" (UID: "c690e8a2-7d89-4924-ba1c-3854eaffd19a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:07.150060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.150035    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:07.151422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.151400    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca" pod "c690e8a2-7d89-4924-ba1c-3854eaffd19a" (UID: "c690e8a2-7d89-4924-ba1c-3854eaffd19a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:07.151668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.151654    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz" pod "c690e8a2-7d89-4924-ba1c-3854eaffd19a" (UID: "c690e8a2-7d89-4924-ba1c-3854eaffd19a"). InnerVolumeSpecName "kube-api-access-jhmbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:07.251237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.251127    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmbz\" (UniqueName: \"kubernetes.io/projected/c690e8a2-7d89-4924-ba1c-3854eaffd19a-kube-api-access-jhmbz\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:07.251237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.251165    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c690e8a2-7d89-4924-ba1c-3854eaffd19a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:07.990451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.990417    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"
Oct 30 22:47:07.990715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.990457    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4" event={"ID":"c690e8a2-7d89-4924-ba1c-3854eaffd19a","Type":"ContainerDied","Data":"8bd08011e538d828b6f2d2ed7b2505886facd67ac766dbc1b299b68da3abf06d"}
Oct 30 22:47:07.990715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:07.990520    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd08011e538d828b6f2d2ed7b2505886facd67ac766dbc1b299b68da3abf06d"
Oct 30 22:47:09.229603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:09.229567    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:11.529033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:11.528982    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:11.697419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:11.697382    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:11.924847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:11.924756    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:12.463984 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:12.463950    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:12.983273 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:12.983239    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"]
Oct 30 22:47:12.983838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:12.983822    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c690e8a2-7d89-4924-ba1c-3854eaffd19a" containerName="revoke-keys"
Oct 30 22:47:12.983891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:12.983838    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c690e8a2-7d89-4924-ba1c-3854eaffd19a" containerName="revoke-keys"
Oct 30 22:47:12.983952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:12.983942    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c690e8a2-7d89-4924-ba1c-3854eaffd19a" containerName="revoke-keys"
Oct 30 22:47:12.988741 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:12.988723    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.008421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.008396    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.008525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.008437    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnmq\" (UniqueName: \"kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.008572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.008548    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.042634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.042601    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"]
Oct 30 22:47:13.085139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.085103    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:13.085139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.085123    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:13.085343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.085126    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:13.085819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.085794    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:47:13.109236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.109210    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.109382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.109290    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.109382 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.109331    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnmq\" (UniqueName: \"kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.109964 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.109943    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.111669 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.111648    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.168786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.168746    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnmq\" (UniqueName: \"kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq\") pod \"maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.302306 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.302225    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:13.685386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.685354    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"]
Oct 30 22:47:13.686186 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:47:13.686158    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec8e241_21e4_43c0_9d97_7215201dd4ef.slice/crio-83e637989fae50b777bd98e164e0a94d3a6e4213f9a53d2baab82ee2d19d6c89 WatchSource:0}: Error finding container 83e637989fae50b777bd98e164e0a94d3a6e4213f9a53d2baab82ee2d19d6c89: Status 404 returned error can't find the container with id 83e637989fae50b777bd98e164e0a94d3a6e4213f9a53d2baab82ee2d19d6c89
Oct 30 22:47:13.988121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:13.988084    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"]
Oct 30 22:47:14.045634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.045596    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql" event={"ID":"4ec8e241-21e4-43c0-9d97-7215201dd4ef","Type":"ContainerStarted","Data":"e2daae6f4c1ffa6444be914dede4a873186a524c8acc9abadbe5a5ebb3bbdf1d"}
Oct 30 22:47:14.045869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.045662    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql" event={"ID":"4ec8e241-21e4-43c0-9d97-7215201dd4ef","Type":"ContainerStarted","Data":"83e637989fae50b777bd98e164e0a94d3a6e4213f9a53d2baab82ee2d19d6c89"}
Oct 30 22:47:14.069763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.069729    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:14.087902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.087864    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"]
Oct 30 22:47:14.323331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.323219    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql" podStartSLOduration=2.323198301 podStartE2EDuration="2.323198301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:14.321029014 +0000 UTC m=+2765.170541989" watchObservedRunningTime="2026-10-30 22:47:14.323198301 +0000 UTC m=+2765.172711274"
Oct 30 22:47:14.338966 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.338928    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:14.487586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:14.487549    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:15.054201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:15.054169    5028 generic.go:356] "Generic (PLEG): container finished" podID="4ec8e241-21e4-43c0-9d97-7215201dd4ef" containerID="e2daae6f4c1ffa6444be914dede4a873186a524c8acc9abadbe5a5ebb3bbdf1d" exitCode=0
Oct 30 22:47:15.054604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:15.054243    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql" event={"ID":"4ec8e241-21e4-43c0-9d97-7215201dd4ef","Type":"ContainerDied","Data":"e2daae6f4c1ffa6444be914dede4a873186a524c8acc9abadbe5a5ebb3bbdf1d"}
Oct 30 22:47:15.645709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:15.645621    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:16.222166 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.222138    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:16.251297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.251266    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca\") pod \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") "
Oct 30 22:47:16.251484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.251319    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq\" (UniqueName: \"kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq\") pod \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") "
Oct 30 22:47:16.251484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.251357    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca\") pod \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\" (UID: \"4ec8e241-21e4-43c0-9d97-7215201dd4ef\") "
Oct 30 22:47:16.251870 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.251841    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca" pod "4ec8e241-21e4-43c0-9d97-7215201dd4ef" (UID: "4ec8e241-21e4-43c0-9d97-7215201dd4ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:16.253587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.253564    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq" pod "4ec8e241-21e4-43c0-9d97-7215201dd4ef" (UID: "4ec8e241-21e4-43c0-9d97-7215201dd4ef"). InnerVolumeSpecName "kube-api-access-7wnmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:16.253587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.253576    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca" pod "4ec8e241-21e4-43c0-9d97-7215201dd4ef" (UID: "4ec8e241-21e4-43c0-9d97-7215201dd4ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:16.352539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.352458    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:16.352797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.352771    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wnmq\" (UniqueName: \"kubernetes.io/projected/4ec8e241-21e4-43c0-9d97-7215201dd4ef-kube-api-access-7wnmq\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:16.352797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:16.352796    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec8e241-21e4-43c0-9d97-7215201dd4ef-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:17.072115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:17.072080    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"
Oct 30 22:47:17.072115 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:17.072096    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql" event={"ID":"4ec8e241-21e4-43c0-9d97-7215201dd4ef","Type":"ContainerDied","Data":"83e637989fae50b777bd98e164e0a94d3a6e4213f9a53d2baab82ee2d19d6c89"}
Oct 30 22:47:17.072352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:17.072142    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e637989fae50b777bd98e164e0a94d3a6e4213f9a53d2baab82ee2d19d6c89"
Oct 30 22:47:17.416455 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:17.416371    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"]
Oct 30 22:47:17.466362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:17.466326    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"]
Oct 30 22:47:17.609392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:17.609363    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:18.757080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:18.757042    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:18.988884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:18.988839    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerName="payload-processing" containerID="cri-o://9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201" gracePeriod=25
Oct 30 22:47:19.088655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:19.088616    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerName="payload-pre-processing" containerID="cri-o://c5675a93a2ebbd1f53ee416611e09a5b08d25439e70bf79dbdd17d1fc4481ffc" gracePeriod=25
Oct 30 22:47:19.734736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:19.734670    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:19.753440 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:19.753406    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:21.057935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:21.057895    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"]
Oct 30 22:47:21.102435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:21.102401    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"]
Oct 30 22:47:21.105601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:21.105566    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10816ca-m8qc9"]
Oct 30 22:47:21.108463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:21.108443    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c7a229-q6qbp"]
Oct 30 22:47:21.291622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:21.291587    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fd5384-7330-4b73-af79-21af15129ca6" path="/var/lib/kubelet/pods/26fd5384-7330-4b73-af79-21af15129ca6/volumes"
Oct 30 22:47:21.292083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:21.292066    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba08d36-492d-4b55-aca5-52d77488e2ec" path="/var/lib/kubelet/pods/5ba08d36-492d-4b55-aca5-52d77488e2ec/volumes"
Oct 30 22:47:23.354991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:23.354955    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:24.406978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:24.406939    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"]
Oct 30 22:47:24.507565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:24.507528    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"]
Oct 30 22:47:24.759027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:24.758996    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:24.912505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:24.912469    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:28.051789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:28.051753    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:28.436616 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:28.436525    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:47:29.407795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.407758    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg" podUID="ce9a00ab-df8f-4571-b99d-1b879a9c323f" containerName="payload-processing" containerID="cri-o://d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698" gracePeriod=25
Oct 30 22:47:29.507976 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.507933    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerName="payload-pre-processing" containerID="cri-o://4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800" gracePeriod=25
Oct 30 22:47:29.672062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.671985    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:47:29.734206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.734170    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:29.753957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.753917    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:29.801623 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.801582    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x\" (UniqueName: \"kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x\") pod \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") "
Oct 30 22:47:29.801860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.801835    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume\") pod \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\" (UID: \"ce9a00ab-df8f-4571-b99d-1b879a9c323f\") "
Oct 30 22:47:29.802231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.802202    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume" pod "ce9a00ab-df8f-4571-b99d-1b879a9c323f" (UID: "ce9a00ab-df8f-4571-b99d-1b879a9c323f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:29.803720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.803676    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x" pod "ce9a00ab-df8f-4571-b99d-1b879a9c323f" (UID: "ce9a00ab-df8f-4571-b99d-1b879a9c323f"). InnerVolumeSpecName "kube-api-access-qxf5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:29.903357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.903320    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9a00ab-df8f-4571-b99d-1b879a9c323f-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:29.903357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:29.903353    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxf5x\" (UniqueName: \"kubernetes.io/projected/ce9a00ab-df8f-4571-b99d-1b879a9c323f-kube-api-access-qxf5x\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:30.180002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.179970    5028 generic.go:356] "Generic (PLEG): container finished" podID="ce9a00ab-df8f-4571-b99d-1b879a9c323f" containerID="d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698" exitCode=0
Oct 30 22:47:30.180189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.180014    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"
Oct 30 22:47:30.180189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.180059    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg" event={"ID":"ce9a00ab-df8f-4571-b99d-1b879a9c323f","Type":"ContainerDied","Data":"d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698"}
Oct 30 22:47:30.180189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.180121    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg" event={"ID":"ce9a00ab-df8f-4571-b99d-1b879a9c323f","Type":"ContainerDied","Data":"d904ce228232eeb85a8adff90018722dca90665b61585d41e697f3fc9f808cfa"}
Oct 30 22:47:30.180189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.180150    5028 scope.go:122] "RemoveContainer" containerID="d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698"
Oct 30 22:47:30.191167 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.191147    5028 scope.go:122] "RemoveContainer" containerID="d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698"
Oct 30 22:47:30.191502 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:30.191479    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698\": container with ID starting with d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698 not found: ID does not exist" containerID="d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698"
Oct 30 22:47:30.191608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.191509    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698"} err="failed to get container status \"d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698\": rpc error: code = NotFound desc = could not find container \"d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698\": container with ID starting with d77cbb82fc3a4ec99d46da32e90046624e39cd2cbbb108a4873ce961bcd1c698 not found: ID does not exist"
Oct 30 22:47:30.344057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.344004    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"]
Oct 30 22:47:30.349050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.349018    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b13755-557c67d777-rqfvg"]
Oct 30 22:47:30.691957 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:30.691924    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:31.290786 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:31.290755    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9a00ab-df8f-4571-b99d-1b879a9c323f" path="/var/lib/kubelet/pods/ce9a00ab-df8f-4571-b99d-1b879a9c323f/volumes"
Oct 30 22:47:32.246877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.246835    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"]
Oct 30 22:47:32.247439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.247423    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4ec8e241-21e4-43c0-9d97-7215201dd4ef" containerName="revoke-keys"
Oct 30 22:47:32.247482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.247440    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec8e241-21e4-43c0-9d97-7215201dd4ef" containerName="revoke-keys"
Oct 30 22:47:32.247482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.247478    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ce9a00ab-df8f-4571-b99d-1b879a9c323f" containerName="payload-processing"
Oct 30 22:47:32.247560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.247484    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9a00ab-df8f-4571-b99d-1b879a9c323f" containerName="payload-processing"
Oct 30 22:47:32.247593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.247571    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec8e241-21e4-43c0-9d97-7215201dd4ef" containerName="revoke-keys"
Oct 30 22:47:32.247593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.247584    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9a00ab-df8f-4571-b99d-1b879a9c323f" containerName="payload-processing"
Oct 30 22:47:32.252435 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.252417    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.288949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.288911    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"]
Oct 30 22:47:32.293088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.293064    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.308479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.308453    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"]
Oct 30 22:47:32.332653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.332627    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:32.332873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.332696    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:32.332873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.332747    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:32.332873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.332768    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:47:32.336272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.336249    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.336430 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.336415    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.336652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.336517    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl29j\" (UniqueName: \"kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.355314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.355281    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"]
Oct 30 22:47:32.437636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.437600    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbnf\" (UniqueName: \"kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.437849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.437722    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" containerID="cri-o://ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93" gracePeriod=45
Oct 30 22:47:32.437849 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.437805    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.437963 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.437859    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.438023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.437954    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl29j\" (UniqueName: \"kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.438484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.438020    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.438484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.438051    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.438718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.438675    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.440551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.440526    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.459863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.459805    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:32.486931 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.486900    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main" containerID="cri-o://c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96" gracePeriod=45
Oct 30 22:47:32.498396 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.498335    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl29j\" (UniqueName: \"kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j\") pod \"maas-api-revoke-sub-3c6ff414-skhl6\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.529794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.529760    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": dial tcp 10.132.0.114:8000: connect: connection refused"
Oct 30 22:47:32.539362 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.539332    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.539472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.539385    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbnf\" (UniqueName: \"kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.539542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.539524    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.540159 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.540136    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.542938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.542002    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.566991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.566956    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:32.600090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.599997    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbnf\" (UniqueName: \"kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf\") pod \"maas-api-revoke-sub-d0c7ae23-nbk9l\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.610092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.610062    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:32.837642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.837611    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:47:32.881838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.881800    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:47:32.943949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.943919    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home\") pod \"fe2ea3ec-c577-4480-b681-6c2af33669fc\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") "
Oct 30 22:47:32.944128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.943959    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home\") pod \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") "
Oct 30 22:47:32.944128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.943989    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir\") pod \"fe2ea3ec-c577-4480-b681-6c2af33669fc\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") "
Oct 30 22:47:32.944128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944039    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache\") pod \"fe2ea3ec-c577-4480-b681-6c2af33669fc\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") "
Oct 30 22:47:32.944128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944061    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2\" (UniqueName: \"kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2\") pod \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") "
Oct 30 22:47:32.944128 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944081    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache\") pod \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") "
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944170    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm\") pod \"fe2ea3ec-c577-4480-b681-6c2af33669fc\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") "
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944209    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home" pod "fe2ea3ec-c577-4480-b681-6c2af33669fc" (UID: "fe2ea3ec-c577-4480-b681-6c2af33669fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944274    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home" pod "65ebfd05-06fb-476e-9d13-30a28e9bfc4a" (UID: "65ebfd05-06fb-476e-9d13-30a28e9bfc4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944318    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir" pod "fe2ea3ec-c577-4480-b681-6c2af33669fc" (UID: "fe2ea3ec-c577-4480-b681-6c2af33669fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944329    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir\") pod \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") "
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944380    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk\" (UniqueName: \"kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk\") pod \"fe2ea3ec-c577-4480-b681-6c2af33669fc\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") "
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944428    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache" pod "fe2ea3ec-c577-4480-b681-6c2af33669fc" (UID: "fe2ea3ec-c577-4480-b681-6c2af33669fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.944444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944445    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs\") pod \"fe2ea3ec-c577-4480-b681-6c2af33669fc\" (UID: \"fe2ea3ec-c577-4480-b681-6c2af33669fc\") "
Oct 30 22:47:32.944877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944432    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache" pod "65ebfd05-06fb-476e-9d13-30a28e9bfc4a" (UID: "65ebfd05-06fb-476e-9d13-30a28e9bfc4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.944877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944482    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs\") pod \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") "
Oct 30 22:47:32.944877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944515    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm\") pod \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\" (UID: \"65ebfd05-06fb-476e-9d13-30a28e9bfc4a\") "
Oct 30 22:47:32.944877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.944555    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir" pod "65ebfd05-06fb-476e-9d13-30a28e9bfc4a" (UID: "65ebfd05-06fb-476e-9d13-30a28e9bfc4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.945087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.945025    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.945087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.945044    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.945087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.945058    5028 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-home\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.945087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.945071    5028 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-tmp-dir\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.945087 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.945083    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.945334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.945097    5028 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-model-cache\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.947158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.947121    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm" pod "fe2ea3ec-c577-4480-b681-6c2af33669fc" (UID: "fe2ea3ec-c577-4480-b681-6c2af33669fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.947319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.947256    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk" pod "fe2ea3ec-c577-4480-b681-6c2af33669fc" (UID: "fe2ea3ec-c577-4480-b681-6c2af33669fc"). InnerVolumeSpecName "kube-api-access-m4pvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:32.947384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.947333    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm" pod "65ebfd05-06fb-476e-9d13-30a28e9bfc4a" (UID: "65ebfd05-06fb-476e-9d13-30a28e9bfc4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:32.947384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.947334    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2" pod "65ebfd05-06fb-476e-9d13-30a28e9bfc4a" (UID: "65ebfd05-06fb-476e-9d13-30a28e9bfc4a"). InnerVolumeSpecName "kube-api-access-rc6f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:32.947384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.947346    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs" pod "65ebfd05-06fb-476e-9d13-30a28e9bfc4a" (UID: "65ebfd05-06fb-476e-9d13-30a28e9bfc4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:32.947384 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.947364    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs" pod "fe2ea3ec-c577-4480-b681-6c2af33669fc" (UID: "fe2ea3ec-c577-4480-b681-6c2af33669fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:32.956093 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:32.956066    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"]
Oct 30 22:47:32.956432 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:47:32.956412    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85beee0_64bb_436c_96ef_0b60708a98f5.slice/crio-bd5b378fb4b03db83235fa01b0a7406f5e49113baf34db8137bdb72669c44a56 WatchSource:0}: Error finding container bd5b378fb4b03db83235fa01b0a7406f5e49113baf34db8137bdb72669c44a56: Status 404 returned error can't find the container with id bd5b378fb4b03db83235fa01b0a7406f5e49113baf34db8137bdb72669c44a56
Oct 30 22:47:33.046568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.046490    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6f2\" (UniqueName: \"kubernetes.io/projected/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-kube-api-access-rc6f2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.046568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.046518    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2ea3ec-c577-4480-b681-6c2af33669fc-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.046568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.046528    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pvk\" (UniqueName: \"kubernetes.io/projected/fe2ea3ec-c577-4480-b681-6c2af33669fc-kube-api-access-m4pvk\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.046568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.046538    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2ea3ec-c577-4480-b681-6c2af33669fc-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.046568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.046548    5028 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-tls-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.046568 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.046558    5028 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ebfd05-06fb-476e-9d13-30a28e9bfc4a-dshm\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.064120 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:47:33.064089    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52308cd1_95c8_4146_a8d7_d6daa7588c28.slice/crio-18daacc06f86f32a042dfac2eaadab2c9ec98287ddcf1dfa21d29a777277f33b WatchSource:0}: Error finding container 18daacc06f86f32a042dfac2eaadab2c9ec98287ddcf1dfa21d29a777277f33b: Status 404 returned error can't find the container with id 18daacc06f86f32a042dfac2eaadab2c9ec98287ddcf1dfa21d29a777277f33b
Oct 30 22:47:33.064351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.064330    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"]
Oct 30 22:47:33.203697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.203646    5028 generic.go:356] "Generic (PLEG): container finished" podID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerID="ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93" exitCode=0
Oct 30 22:47:33.203872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.203750    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"
Oct 30 22:47:33.203872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.203756    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" event={"ID":"fe2ea3ec-c577-4480-b681-6c2af33669fc","Type":"ContainerDied","Data":"ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93"}
Oct 30 22:47:33.203872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.203837    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg" event={"ID":"fe2ea3ec-c577-4480-b681-6c2af33669fc","Type":"ContainerDied","Data":"792934417ddb1ba64dc6c4e29c5bdde056f4100f4e621308e437ba381a21a781"}
Oct 30 22:47:33.203872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.203863    5028 scope.go:122] "RemoveContainer" containerID="ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93"
Oct 30 22:47:33.205448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.205420    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l" event={"ID":"52308cd1-95c8-4146-a8d7-d6daa7588c28","Type":"ContainerStarted","Data":"18daacc06f86f32a042dfac2eaadab2c9ec98287ddcf1dfa21d29a777277f33b"}
Oct 30 22:47:33.207221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.207180    5028 generic.go:356] "Generic (PLEG): container finished" podID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerID="c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96" exitCode=0
Oct 30 22:47:33.207221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.207210    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" event={"ID":"65ebfd05-06fb-476e-9d13-30a28e9bfc4a","Type":"ContainerDied","Data":"c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96"}
Oct 30 22:47:33.207376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.207265    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n" event={"ID":"65ebfd05-06fb-476e-9d13-30a28e9bfc4a","Type":"ContainerDied","Data":"375e81ea9af305af4e7abbc952df67838909769237cb1074508aeb4d6ebae12f"}
Oct 30 22:47:33.207376 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.207219    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"
Oct 30 22:47:33.208718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.208696    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6" event={"ID":"b85beee0-64bb-436c-96ef-0b60708a98f5","Type":"ContainerStarted","Data":"bd5b378fb4b03db83235fa01b0a7406f5e49113baf34db8137bdb72669c44a56"}
Oct 30 22:47:33.216073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.216053    5028 scope.go:122] "RemoveContainer" containerID="ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93"
Oct 30 22:47:33.216382 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:33.216361    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93\": container with ID starting with ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93 not found: ID does not exist" containerID="ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93"
Oct 30 22:47:33.216467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.216388    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93"} err="failed to get container status \"ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93\": rpc error: code = NotFound desc = could not find container \"ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93\": container with ID starting with ef499861f2da1a5cc1449d294e693737d903dcdbffc28c907737147386a49a93 not found: ID does not exist"
Oct 30 22:47:33.216467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.216406    5028 scope.go:122] "RemoveContainer" containerID="c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96"
Oct 30 22:47:33.232005 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.231857    5028 scope.go:122] "RemoveContainer" containerID="c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96"
Oct 30 22:47:33.232424 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:33.232345    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96\": container with ID starting with c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96 not found: ID does not exist" containerID="c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96"
Oct 30 22:47:33.232424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.232388    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96"} err="failed to get container status \"c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96\": rpc error: code = NotFound desc = could not find container \"c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96\": container with ID starting with c2560a1599593f4f07469eca6142ecc5acf3514037a188c9fefd74cd83b7cc96 not found: ID does not exist"
Oct 30 22:47:33.382270 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.382178    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"]
Oct 30 22:47:33.389676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.389645    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-2-w4-b13755-kserve-545d9ff47f-954dg"]
Oct 30 22:47:33.498132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.498094    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"]
Oct 30 22:47:33.503097 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.503071    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/premium-w4-b13755-kserve-6bfdbf589-6kz2n"]
Oct 30 22:47:33.906860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:33.906829    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.152604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.152520    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.228925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.221902    5028 generic.go:356] "Generic (PLEG): container finished" podID="52308cd1-95c8-4146-a8d7-d6daa7588c28" containerID="b4d35d13d302bd49b262bb91e6f30cc5ddeacf10223e49054352e238ec780a4c" exitCode=0
Oct 30 22:47:34.228925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.222054    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l" event={"ID":"52308cd1-95c8-4146-a8d7-d6daa7588c28","Type":"ContainerDied","Data":"b4d35d13d302bd49b262bb91e6f30cc5ddeacf10223e49054352e238ec780a4c"}
Oct 30 22:47:34.238794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.238744    5028 generic.go:356] "Generic (PLEG): container finished" podID="b85beee0-64bb-436c-96ef-0b60708a98f5" containerID="4deca31b7339d0edf4ffcc2fbe1f27a427f0b134e83f8124e5bd2f1d34bfd162" exitCode=0
Oct 30 22:47:34.238980 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.238805    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6" event={"ID":"b85beee0-64bb-436c-96ef-0b60708a98f5","Type":"ContainerDied","Data":"4deca31b7339d0edf4ffcc2fbe1f27a427f0b134e83f8124e5bd2f1d34bfd162"}
Oct 30 22:47:34.315929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.315899    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.384550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.384520    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"]
Oct 30 22:47:34.387630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.387607    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb3406-mkzt4"]
Oct 30 22:47:34.831560 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:34.831524    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:35.296625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.296586    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" path="/var/lib/kubelet/pods/65ebfd05-06fb-476e-9d13-30a28e9bfc4a/volumes"
Oct 30 22:47:35.297094 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.297077    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c690e8a2-7d89-4924-ba1c-3854eaffd19a" path="/var/lib/kubelet/pods/c690e8a2-7d89-4924-ba1c-3854eaffd19a/volumes"
Oct 30 22:47:35.297474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.297461    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" path="/var/lib/kubelet/pods/fe2ea3ec-c577-4480-b681-6c2af33669fc/volumes"
Oct 30 22:47:35.409083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.409054    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:35.443070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.443045    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:35.484442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.484410    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca\") pod \"52308cd1-95c8-4146-a8d7-d6daa7588c28\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") "
Oct 30 22:47:35.484595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.484542    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf\" (UniqueName: \"kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf\") pod \"52308cd1-95c8-4146-a8d7-d6daa7588c28\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") "
Oct 30 22:47:35.484595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.484591    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca\") pod \"52308cd1-95c8-4146-a8d7-d6daa7588c28\" (UID: \"52308cd1-95c8-4146-a8d7-d6daa7588c28\") "
Oct 30 22:47:35.484873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.484841    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca" pod "52308cd1-95c8-4146-a8d7-d6daa7588c28" (UID: "52308cd1-95c8-4146-a8d7-d6daa7588c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:35.486661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.486640    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca" pod "52308cd1-95c8-4146-a8d7-d6daa7588c28" (UID: "52308cd1-95c8-4146-a8d7-d6daa7588c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:35.486773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.486736    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf" pod "52308cd1-95c8-4146-a8d7-d6daa7588c28" (UID: "52308cd1-95c8-4146-a8d7-d6daa7588c28"). InnerVolumeSpecName "kube-api-access-jsbnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:35.585990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.585910    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca\") pod \"b85beee0-64bb-436c-96ef-0b60708a98f5\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") "
Oct 30 22:47:35.586140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.586115    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca\") pod \"b85beee0-64bb-436c-96ef-0b60708a98f5\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") "
Oct 30 22:47:35.586193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.586144    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j\" (UniqueName: \"kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j\") pod \"b85beee0-64bb-436c-96ef-0b60708a98f5\" (UID: \"b85beee0-64bb-436c-96ef-0b60708a98f5\") "
Oct 30 22:47:35.586412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.586392    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:35.586412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.586409    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbnf\" (UniqueName: \"kubernetes.io/projected/52308cd1-95c8-4146-a8d7-d6daa7588c28-kube-api-access-jsbnf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:35.586412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.586419    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52308cd1-95c8-4146-a8d7-d6daa7588c28-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:35.586711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.586430    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca" pod "b85beee0-64bb-436c-96ef-0b60708a98f5" (UID: "b85beee0-64bb-436c-96ef-0b60708a98f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:35.588022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.587995    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca" pod "b85beee0-64bb-436c-96ef-0b60708a98f5" (UID: "b85beee0-64bb-436c-96ef-0b60708a98f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:35.588169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.588148    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j" pod "b85beee0-64bb-436c-96ef-0b60708a98f5" (UID: "b85beee0-64bb-436c-96ef-0b60708a98f5"). InnerVolumeSpecName "kube-api-access-rl29j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:35.687432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.687397    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:35.687432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.687426    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl29j\" (UniqueName: \"kubernetes.io/projected/b85beee0-64bb-436c-96ef-0b60708a98f5-kube-api-access-rl29j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:35.687432 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:35.687436    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85beee0-64bb-436c-96ef-0b60708a98f5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:36.256064 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.256031    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"
Oct 30 22:47:36.256259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.256032    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l" event={"ID":"52308cd1-95c8-4146-a8d7-d6daa7588c28","Type":"ContainerDied","Data":"18daacc06f86f32a042dfac2eaadab2c9ec98287ddcf1dfa21d29a777277f33b"}
Oct 30 22:47:36.256259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.256166    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18daacc06f86f32a042dfac2eaadab2c9ec98287ddcf1dfa21d29a777277f33b"
Oct 30 22:47:36.257422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.257405    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"
Oct 30 22:47:36.257551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.257403    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6" event={"ID":"b85beee0-64bb-436c-96ef-0b60708a98f5","Type":"ContainerDied","Data":"bd5b378fb4b03db83235fa01b0a7406f5e49113baf34db8137bdb72669c44a56"}
Oct 30 22:47:36.257551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.257537    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5b378fb4b03db83235fa01b0a7406f5e49113baf34db8137bdb72669c44a56"
Oct 30 22:47:36.916795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:36.916743    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:38.928230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:38.928161    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:39.260824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.260793    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:47:39.283956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.283917    5028 generic.go:356] "Generic (PLEG): container finished" podID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerID="c5675a93a2ebbd1f53ee416611e09a5b08d25439e70bf79dbdd17d1fc4481ffc" exitCode=0
Oct 30 22:47:39.285774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.285747    5028 generic.go:356] "Generic (PLEG): container finished" podID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerID="9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201" exitCode=0
Oct 30 22:47:39.285919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.285816    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"
Oct 30 22:47:39.295073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.295049    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" event={"ID":"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e","Type":"ContainerDied","Data":"c5675a93a2ebbd1f53ee416611e09a5b08d25439e70bf79dbdd17d1fc4481ffc"}
Oct 30 22:47:39.295219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.295097    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" event={"ID":"bbbb8c2d-a984-48eb-8788-b54a17296b37","Type":"ContainerDied","Data":"9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201"}
Oct 30 22:47:39.295219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.295130    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s" event={"ID":"bbbb8c2d-a984-48eb-8788-b54a17296b37","Type":"ContainerDied","Data":"068d1d4f14f3c4505ee034e1b633a57f0090a3d8dc1f8bffb3ec003ac10fd4a1"}
Oct 30 22:47:39.295219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.295159    5028 scope.go:122] "RemoveContainer" containerID="9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201"
Oct 30 22:47:39.308209 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.308185    5028 scope.go:122] "RemoveContainer" containerID="9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201"
Oct 30 22:47:39.308558 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:39.308536    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201\": container with ID starting with 9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201 not found: ID does not exist" containerID="9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201"
Oct 30 22:47:39.308655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.308566    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201"} err="failed to get container status \"9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201\": rpc error: code = NotFound desc = could not find container \"9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201\": container with ID starting with 9ed652bfd94331ef709503ea77c12b138419f134467b7008c70877b304fb7201 not found: ID does not exist"
Oct 30 22:47:39.360630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.360603    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:47:39.429599 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.429568    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv\" (UniqueName: \"kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv\") pod \"bbbb8c2d-a984-48eb-8788-b54a17296b37\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") "
Oct 30 22:47:39.429837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.429664    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume\") pod \"bbbb8c2d-a984-48eb-8788-b54a17296b37\" (UID: \"bbbb8c2d-a984-48eb-8788-b54a17296b37\") "
Oct 30 22:47:39.430123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.430087    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume" pod "bbbb8c2d-a984-48eb-8788-b54a17296b37" (UID: "bbbb8c2d-a984-48eb-8788-b54a17296b37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:39.431930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.431907    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv" pod "bbbb8c2d-a984-48eb-8788-b54a17296b37" (UID: "bbbb8c2d-a984-48eb-8788-b54a17296b37"). InnerVolumeSpecName "kube-api-access-kxckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:39.530792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.530757    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume\") pod \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") "
Oct 30 22:47:39.531016 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.530814    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4\" (UniqueName: \"kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4\") pod \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\" (UID: \"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e\") "
Oct 30 22:47:39.531155 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.531131    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume" pod "3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" (UID: "3b6059c1-b900-4cb8-9955-9aa6fc40bb5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:39.531346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.531327    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:39.531412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.531350    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxckv\" (UniqueName: \"kubernetes.io/projected/bbbb8c2d-a984-48eb-8788-b54a17296b37-kube-api-access-kxckv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:39.531412 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.531365    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbb8c2d-a984-48eb-8788-b54a17296b37-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:39.533075 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.533050    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4" pod "3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" (UID: "3b6059c1-b900-4cb8-9955-9aa6fc40bb5e"). InnerVolumeSpecName "kube-api-access-pj4v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:39.632676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.632641    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4v4\" (UniqueName: \"kubernetes.io/projected/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e-kube-api-access-pj4v4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:39.703221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.703181    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"]
Oct 30 22:47:39.710462 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:39.710433    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb3406-896f479f9-ptx5s"]
Oct 30 22:47:40.297170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:40.297081    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl" event={"ID":"3b6059c1-b900-4cb8-9955-9aa6fc40bb5e","Type":"ContainerDied","Data":"2cc09b19fc6d19e519f5985396ce15775241995a74b969aa65bedc147075a052"}
Oct 30 22:47:40.297170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:40.297114    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"
Oct 30 22:47:40.297170 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:40.297148    5028 scope.go:122] "RemoveContainer" containerID="c5675a93a2ebbd1f53ee416611e09a5b08d25439e70bf79dbdd17d1fc4481ffc"
Oct 30 22:47:40.457164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:40.457137    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"]
Oct 30 22:47:40.463099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:40.463074    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb3406-57867f4bc5-f6cnl"]
Oct 30 22:47:41.292193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:41.292158    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" path="/var/lib/kubelet/pods/3b6059c1-b900-4cb8-9955-9aa6fc40bb5e/volumes"
Oct 30 22:47:41.292790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:41.292772    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" path="/var/lib/kubelet/pods/bbbb8c2d-a984-48eb-8788-b54a17296b37/volumes"
Oct 30 22:47:41.403069 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:41.403029    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:41.552348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:41.552264    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:41.776169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:41.776126    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:42.602837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.602801    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"]
Oct 30 22:47:42.604515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.604490    5028 cpu_manager.go:414] "containerMap: removing container" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main"
Oct 30 22:47:42.604721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.604705    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main"
Oct 30 22:47:42.604843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.604832    5028 cpu_manager.go:414] "containerMap: removing container" podUID="52308cd1-95c8-4146-a8d7-d6daa7588c28" containerName="revoke-keys"
Oct 30 22:47:42.604927 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.604916    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52308cd1-95c8-4146-a8d7-d6daa7588c28" containerName="revoke-keys"
Oct 30 22:47:42.605125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.605111    5028 cpu_manager.go:414] "containerMap: removing container" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerName="payload-processing"
Oct 30 22:47:42.605254 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.605216    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerName="payload-processing"
Oct 30 22:47:42.605365 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.605356    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main"
Oct 30 22:47:42.605492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.605434    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main"
Oct 30 22:47:42.619709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.605579    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b85beee0-64bb-436c-96ef-0b60708a98f5" containerName="revoke-keys"
Oct 30 22:47:42.619959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.619932    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85beee0-64bb-436c-96ef-0b60708a98f5" containerName="revoke-keys"
Oct 30 22:47:42.620114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620101    5028 cpu_manager.go:414] "containerMap: removing container" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerName="payload-pre-processing"
Oct 30 22:47:42.620202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620190    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerName="payload-pre-processing"
Oct 30 22:47:42.620506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620489    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2ea3ec-c577-4480-b681-6c2af33669fc" containerName="main"
Oct 30 22:47:42.620616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620605    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85beee0-64bb-436c-96ef-0b60708a98f5" containerName="revoke-keys"
Oct 30 22:47:42.620738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620727    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6059c1-b900-4cb8-9955-9aa6fc40bb5e" containerName="payload-pre-processing"
Oct 30 22:47:42.620822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620814    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ebfd05-06fb-476e-9d13-30a28e9bfc4a" containerName="main"
Oct 30 22:47:42.620902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620893    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="52308cd1-95c8-4146-a8d7-d6daa7588c28" containerName="revoke-keys"
Oct 30 22:47:42.620981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.620972    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbb8c2d-a984-48eb-8788-b54a17296b37" containerName="payload-processing"
Oct 30 22:47:42.630717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.626600    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.659374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.659338    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"]
Oct 30 22:47:42.691626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.691590    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:42.691902 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.691613    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.692117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.691616    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:47:42.692117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.691829    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:42.692336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.692321    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.693588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.693553    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hxb\" (UniqueName: \"kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.734174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.734141    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:42.794726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.794653    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.794940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.794752    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.794940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.794815    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hxb\" (UniqueName: \"kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.795512 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.795484    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.797718 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.797657    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.854268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.854189    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hxb\" (UniqueName: \"kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb\") pod \"maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:42.974013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:42.973977    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:43.364415 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:47:43.364382    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832d6f33_54ba_43cd_8e81_be6bac55e067.slice/crio-94ecbcc3dc0852c4eebea45d97b90e387575302a5363eeec46cbd753fe0aa90b WatchSource:0}: Error finding container 94ecbcc3dc0852c4eebea45d97b90e387575302a5363eeec46cbd753fe0aa90b: Status 404 returned error can't find the container with id 94ecbcc3dc0852c4eebea45d97b90e387575302a5363eeec46cbd753fe0aa90b
Oct 30 22:47:43.365409 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:43.365377    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"]
Oct 30 22:47:43.589082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:43.589046    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:44.334120 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:44.334080    5028 generic.go:356] "Generic (PLEG): container finished" podID="832d6f33-54ba-43cd-8e81-be6bac55e067" containerID="34423e18a2eca2b9010fb4bd63c8e52d776b41a0c65dc64f942ddff827153824" exitCode=0
Oct 30 22:47:44.334627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:44.334159    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q" event={"ID":"832d6f33-54ba-43cd-8e81-be6bac55e067","Type":"ContainerDied","Data":"34423e18a2eca2b9010fb4bd63c8e52d776b41a0c65dc64f942ddff827153824"}
Oct 30 22:47:44.334627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:44.334216    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q" event={"ID":"832d6f33-54ba-43cd-8e81-be6bac55e067","Type":"ContainerStarted","Data":"94ecbcc3dc0852c4eebea45d97b90e387575302a5363eeec46cbd753fe0aa90b"}
Oct 30 22:47:45.508967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.508944    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:45.525038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.525012    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca\") pod \"832d6f33-54ba-43cd-8e81-be6bac55e067\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") "
Oct 30 22:47:45.525169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.525076    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca\") pod \"832d6f33-54ba-43cd-8e81-be6bac55e067\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") "
Oct 30 22:47:45.525169 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.525120    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb\" (UniqueName: \"kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb\") pod \"832d6f33-54ba-43cd-8e81-be6bac55e067\" (UID: \"832d6f33-54ba-43cd-8e81-be6bac55e067\") "
Oct 30 22:47:45.525353 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.525331    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca" pod "832d6f33-54ba-43cd-8e81-be6bac55e067" (UID: "832d6f33-54ba-43cd-8e81-be6bac55e067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:45.525588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.525564    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.527182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.527156    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca" pod "832d6f33-54ba-43cd-8e81-be6bac55e067" (UID: "832d6f33-54ba-43cd-8e81-be6bac55e067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:45.527358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.527331    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb" pod "832d6f33-54ba-43cd-8e81-be6bac55e067" (UID: "832d6f33-54ba-43cd-8e81-be6bac55e067"). InnerVolumeSpecName "kube-api-access-96hxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:45.626316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.626233    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832d6f33-54ba-43cd-8e81-be6bac55e067-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.626316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:45.626269    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hxb\" (UniqueName: \"kubernetes.io/projected/832d6f33-54ba-43cd-8e81-be6bac55e067-kube-api-access-96hxb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.351133 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:46.351099    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"
Oct 30 22:47:46.351325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:46.351128    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q" event={"ID":"832d6f33-54ba-43cd-8e81-be6bac55e067","Type":"ContainerDied","Data":"94ecbcc3dc0852c4eebea45d97b90e387575302a5363eeec46cbd753fe0aa90b"}
Oct 30 22:47:46.351325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:46.351177    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94ecbcc3dc0852c4eebea45d97b90e387575302a5363eeec46cbd753fe0aa90b"
Oct 30 22:47:46.917062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:46.917021    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:48.590222 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:48.590185    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:49.769516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.769455    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:47:49.869049 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.869017    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume\") pod \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") "
Oct 30 22:47:49.869235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.869113    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk\" (UniqueName: \"kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk\") pod \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\" (UID: \"08508b9b-d3ed-42c8-aedd-13aaad6cb943\") "
Oct 30 22:47:49.869423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.869398    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume" pod "08508b9b-d3ed-42c8-aedd-13aaad6cb943" (UID: "08508b9b-d3ed-42c8-aedd-13aaad6cb943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:49.871298 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.871266    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk" pod "08508b9b-d3ed-42c8-aedd-13aaad6cb943" (UID: "08508b9b-d3ed-42c8-aedd-13aaad6cb943"). InnerVolumeSpecName "kube-api-access-7nrhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:49.970616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.970567    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08508b9b-d3ed-42c8-aedd-13aaad6cb943-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.970616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:49.970611    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrhk\" (UniqueName: \"kubernetes.io/projected/08508b9b-d3ed-42c8-aedd-13aaad6cb943-kube-api-access-7nrhk\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:50.391804 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.391772    5028 generic.go:356] "Generic (PLEG): container finished" podID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerID="4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800" exitCode=0
Oct 30 22:47:50.391992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.391830    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"
Oct 30 22:47:50.391992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.391859    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" event={"ID":"08508b9b-d3ed-42c8-aedd-13aaad6cb943","Type":"ContainerDied","Data":"4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800"}
Oct 30 22:47:50.391992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.391913    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl" event={"ID":"08508b9b-d3ed-42c8-aedd-13aaad6cb943","Type":"ContainerDied","Data":"9e8b7d6387a350694daf04eb0e2f24f68407b464cdff31c61ad29e2c1f203557"}
Oct 30 22:47:50.391992 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.391946    5028 scope.go:122] "RemoveContainer" containerID="4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800"
Oct 30 22:47:50.403338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.403319    5028 scope.go:122] "RemoveContainer" containerID="4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800"
Oct 30 22:47:50.403617 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:50.403586    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800\": container with ID starting with 4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800 not found: ID does not exist" containerID="4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800"
Oct 30 22:47:50.403666 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.403632    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800"} err="failed to get container status \"4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800\": rpc error: code = NotFound desc = could not find container \"4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800\": container with ID starting with 4befb28fc9d9a481418c80524808dc32bd0a45fa2652ee40f4d9292ed5f22800 not found: ID does not exist"
Oct 30 22:47:50.551970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.551940    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"]
Oct 30 22:47:50.598591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:50.598556    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b13755-6cb84c449f-bplzl"]
Oct 30 22:47:51.290519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:51.290485    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" path="/var/lib/kubelet/pods/08508b9b-d3ed-42c8-aedd-13aaad6cb943/volumes"
Oct 30 22:47:53.357380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:53.357333    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:53.899102 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:53.899054    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:53.965098 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:53.965061    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"]
Oct 30 22:47:53.965339 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:53.965311    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy" containerID="cri-o://56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524" gracePeriod=30
Oct 30 22:47:54.011346 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:54.011308    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"]
Oct 30 22:47:54.011559 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:54.011509    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy" containerID="cri-o://59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39" gracePeriod=30
Oct 30 22:47:54.067402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:54.067360    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:54.548300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:54.548263    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:54.715783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:54.715750    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:56.518561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:56.518524    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:57.476652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:57.476615    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:58.436506 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:58.436469    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:47:58.545965 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:58.545932    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:58.903609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:58.903563    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:59.191872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.191814    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:59.282728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.282696    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:47:59.322395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.322366    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:47:59.386595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386540    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"]
Oct 30 22:47:59.386842 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386820    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.386895 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386858    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.386956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386893    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.386956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386923    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.386956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386948    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.387129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.386983    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.387129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387013    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.387129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387044    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.387129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387111    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387138    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387168    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387243    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387285    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387316    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387310    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.387355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387322    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:59.387754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387557    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.387754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387590    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.387754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387637    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.387928 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387814    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.387996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387918    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.387996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.387927    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:59.388221 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.388196    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.388300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.388217    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.388300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.388256    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.388416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.388327    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy\") pod \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\" (UID: \"2d8720c1-5f98-4e9a-acc8-a65378d1b743\") "
Oct 30 22:47:59.388416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.388384    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs\") pod \"78d3a0dc-cc45-4619-bb5b-da11755ba935\" (UID: \"78d3a0dc-cc45-4619-bb5b-da11755ba935\") "
Oct 30 22:47:59.389027 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.388995    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389029    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389047    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d3a0dc-cc45-4619-bb5b-da11755ba935-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389063    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389078    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389094    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389108    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389123    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.389140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.389137    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.390986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.390956    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "kube-api-access-thkb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:59.392088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.391768    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:59.392088 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.391821    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b13755-7tcql"]
Oct 30 22:47:59.392286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.392124    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.392601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.392574    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.392726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.392581    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo" pod "78d3a0dc-cc45-4619-bb5b-da11755ba935" (UID: "78d3a0dc-cc45-4619-bb5b-da11755ba935"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:59.392823 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.392769    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "kube-api-access-gh894". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:59.392891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.392833    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:59.393047 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.393027    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:59.394055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.394035    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy" pod "2d8720c1-5f98-4e9a-acc8-a65378d1b743" (UID: "2d8720c1-5f98-4e9a-acc8-a65378d1b743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:59.472269 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.472236    5028 generic.go:356] "Generic (PLEG): container finished" podID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerID="59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39" exitCode=0
Oct 30 22:47:59.472796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.472290    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"
Oct 30 22:47:59.472796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.472328    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" event={"ID":"2d8720c1-5f98-4e9a-acc8-a65378d1b743","Type":"ContainerDied","Data":"59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39"}
Oct 30 22:47:59.472796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.472397    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw" event={"ID":"2d8720c1-5f98-4e9a-acc8-a65378d1b743","Type":"ContainerDied","Data":"74371c85b4ebd1003fee8ef28bc84db89decfed836ac7d55fca804ad8543aeb2"}
Oct 30 22:47:59.472796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.472430    5028 scope.go:122] "RemoveContainer" containerID="59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39"
Oct 30 22:47:59.474180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.474155    5028 generic.go:356] "Generic (PLEG): container finished" podID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerID="56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524" exitCode=0
Oct 30 22:47:59.474392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.474211    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"
Oct 30 22:47:59.474497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.474228    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" event={"ID":"78d3a0dc-cc45-4619-bb5b-da11755ba935","Type":"ContainerDied","Data":"56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524"}
Oct 30 22:47:59.474565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.474523    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88" event={"ID":"78d3a0dc-cc45-4619-bb5b-da11755ba935","Type":"ContainerDied","Data":"af78e0231d20a9560c7dad6a32cc34e28310a8951afa7ff66380ab204e0c3c37"}
Oct 30 22:47:59.486554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.486532    5028 scope.go:122] "RemoveContainer" containerID="59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39"
Oct 30 22:47:59.486877 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:59.486854    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39\": container with ID starting with 59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39 not found: ID does not exist" containerID="59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39"
Oct 30 22:47:59.486977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.486888    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39"} err="failed to get container status \"59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39\": rpc error: code = NotFound desc = could not find container \"59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39\": container with ID starting with 59f9306cd11823cc7d7f4adfe7ab0b1f18103897340eb41467d6ee5a8bf56b39 not found: ID does not exist"
Oct 30 22:47:59.486977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.486911    5028 scope.go:122] "RemoveContainer" containerID="56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524"
Oct 30 22:47:59.490479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490457    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490482    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490497    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490512    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thkb6\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-kube-api-access-thkb6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490526    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh894\" (UniqueName: \"kubernetes.io/projected/2d8720c1-5f98-4e9a-acc8-a65378d1b743-kube-api-access-gh894\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490539    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490552    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8720c1-5f98-4e9a-acc8-a65378d1b743-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490566    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.490641 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.490578    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d3a0dc-cc45-4619-bb5b-da11755ba935-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:59.503207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.503187    5028 scope.go:122] "RemoveContainer" containerID="56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524"
Oct 30 22:47:59.503560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:47:59.503542    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524\": container with ID starting with 56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524 not found: ID does not exist" containerID="56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524"
Oct 30 22:47:59.503631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.503571    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524"} err="failed to get container status \"56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524\": rpc error: code = NotFound desc = could not find container \"56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524\": container with ID starting with 56dbe0bb591c8fee71b31381e62d4a228842a28357995aca1557cfb1fb962524 not found: ID does not exist"
Oct 30 22:47:59.676210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.676172    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"]
Oct 30 22:47:59.682745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.682709    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-9qn88"]
Oct 30 22:47:59.789373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.789287    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"]
Oct 30 22:47:59.797434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.797400    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zldcw"]
Oct 30 22:47:59.854894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:47:59.854853    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:00.120214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:00.120124    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:00.435156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:00.435061    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:00.592851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:00.592822    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:00.730135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:00.730093    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"]
Oct 30 22:48:00.733652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:00.733610    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-afb98c-m6k2q"]
Oct 30 22:48:01.164482 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:01.164401    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:01.291127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:01.291091    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" path="/var/lib/kubelet/pods/2d8720c1-5f98-4e9a-acc8-a65378d1b743/volumes"
Oct 30 22:48:01.291543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:01.291528    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec8e241-21e4-43c0-9d97-7215201dd4ef" path="/var/lib/kubelet/pods/4ec8e241-21e4-43c0-9d97-7215201dd4ef/volumes"
Oct 30 22:48:01.291982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:01.291969    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" path="/var/lib/kubelet/pods/78d3a0dc-cc45-4619-bb5b-da11755ba935/volumes"
Oct 30 22:48:01.292674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:01.292659    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832d6f33-54ba-43cd-8e81-be6bac55e067" path="/var/lib/kubelet/pods/832d6f33-54ba-43cd-8e81-be6bac55e067/volumes"
Oct 30 22:48:03.110129 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:03.110075    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:03.262959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:03.262925    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:05.251697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:05.251664    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:06.292140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:06.292103    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:06.450033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:06.449994    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:06.634001 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:06.633914    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"]
Oct 30 22:48:06.634266 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:06.634238    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" containerID="cri-o://7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb" gracePeriod=30
Oct 30 22:48:06.648139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:06.648093    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:08.235890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:08.235855    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:08.396474 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:08.396435    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:08.567360 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:08.567282    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"]
Oct 30 22:48:08.567524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:08.567486    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" containerID="cri-o://53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae" gracePeriod=30
Oct 30 22:48:08.583639 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:08.583608    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:09.130204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.130163    5028 patch_prober.go:29] interesting pod/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:48:09.130386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.130217    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:48:09.383168 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383090    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"]
Oct 30 22:48:09.383583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383569    5028 cpu_manager.go:414] "containerMap: removing container" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy"
Oct 30 22:48:09.383629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383583    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy"
Oct 30 22:48:09.383629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383611    5028 cpu_manager.go:414] "containerMap: removing container" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerName="payload-pre-processing"
Oct 30 22:48:09.383629 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383616    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerName="payload-pre-processing"
Oct 30 22:48:09.383747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383636    5028 cpu_manager.go:414] "containerMap: removing container" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy"
Oct 30 22:48:09.383747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383642    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy"
Oct 30 22:48:09.383747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383650    5028 cpu_manager.go:414] "containerMap: removing container" podUID="832d6f33-54ba-43cd-8e81-be6bac55e067" containerName="revoke-keys"
Oct 30 22:48:09.383747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383655    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832d6f33-54ba-43cd-8e81-be6bac55e067" containerName="revoke-keys"
Oct 30 22:48:09.383747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383740    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="08508b9b-d3ed-42c8-aedd-13aaad6cb943" containerName="payload-pre-processing"
Oct 30 22:48:09.383901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383750    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8720c1-5f98-4e9a-acc8-a65378d1b743" containerName="istio-proxy"
Oct 30 22:48:09.383901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383755    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="832d6f33-54ba-43cd-8e81-be6bac55e067" containerName="revoke-keys"
Oct 30 22:48:09.383901 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.383763    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d3a0dc-cc45-4619-bb5b-da11755ba935" containerName="istio-proxy"
Oct 30 22:48:09.386903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.386886    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.437734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.437652    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:09.437734 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.437734    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:48:09.438541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.438127    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:09.441370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.441347    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"]
Oct 30 22:48:09.484670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.484634    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:09.501033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.501001    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.501234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.501049    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjln\" (UniqueName: \"kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.501234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.501120    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.602806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.602765    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.602806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.602810    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjln\" (UniqueName: \"kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.603040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.602830    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.603519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.603491    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.605194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.605171    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.662137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.662056    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjln\" (UniqueName: \"kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.701320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.701288    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:09.916367 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:09.916188    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.071573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:10.071541    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"]
Oct 30 22:48:10.072046 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:48:10.072023    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384b2655_148d_493b_9d3a_88affe878640.slice/crio-2c6ba95d36c95bc6d356d5f702e13df63e38202f34e076e115aec0050ad84283 WatchSource:0}: Error finding container 2c6ba95d36c95bc6d356d5f702e13df63e38202f34e076e115aec0050ad84283: Status 404 returned error can't find the container with id 2c6ba95d36c95bc6d356d5f702e13df63e38202f34e076e115aec0050ad84283
Oct 30 22:48:10.556805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:10.556708    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5" event={"ID":"384b2655-148d-493b-9d3a-88affe878640","Type":"ContainerStarted","Data":"64abe0da11e6003ad4b74a804956cb0fcbc30ed0e87099e893c433bfbca8b09c"}
Oct 30 22:48:10.556805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:10.556775    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5" event={"ID":"384b2655-148d-493b-9d3a-88affe878640","Type":"ContainerStarted","Data":"2c6ba95d36c95bc6d356d5f702e13df63e38202f34e076e115aec0050ad84283"}
Oct 30 22:48:10.671939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:10.671885    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5" podStartSLOduration=1.671870948 podStartE2EDuration="1.671870948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:10.671157482 +0000 UTC m=+2821.520670456" watchObservedRunningTime="2026-10-30 22:48:10.671870948 +0000 UTC m=+2821.521383921"
Oct 30 22:48:11.242112 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:11.242076    5028 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:48:11.242319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:11.242158    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:48:11.573585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:11.573481    5028 generic.go:356] "Generic (PLEG): container finished" podID="384b2655-148d-493b-9d3a-88affe878640" containerID="64abe0da11e6003ad4b74a804956cb0fcbc30ed0e87099e893c433bfbca8b09c" exitCode=0
Oct 30 22:48:11.573585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:11.573566    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5" event={"ID":"384b2655-148d-493b-9d3a-88affe878640","Type":"ContainerDied","Data":"64abe0da11e6003ad4b74a804956cb0fcbc30ed0e87099e893c433bfbca8b09c"}
Oct 30 22:48:11.891836 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:11.891812    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:48:12.028439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028400    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.028627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028455    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.028627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028494    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.028627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028539    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.028627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028587    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.028929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028647    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.028929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028744    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:12.028929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028776    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:12.028929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028779    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.029154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028943    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.029154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029007    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token\") pod \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\" (UID: \"c229eb45-6e20-4d24-bb47-dc47daa3dc0a\") "
Oct 30 22:48:12.029154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028940    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:12.029154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029068    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:12.029154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.028980    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:12.029477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029452    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.029576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029482    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.029576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029498    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.029576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029512    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.029576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.029527    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.030978 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.030953    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "kube-api-access-s2pgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:12.031150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.031118    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:12.031308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.031289    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:12.031427 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.031406    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token" pod "c229eb45-6e20-4d24-bb47-dc47daa3dc0a" (UID: "c229eb45-6e20-4d24-bb47-dc47daa3dc0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:12.130586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.130492    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.130586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.130532    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.130586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.130546    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.130586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.130559    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2pgr\" (UniqueName: \"kubernetes.io/projected/c229eb45-6e20-4d24-bb47-dc47daa3dc0a-kube-api-access-s2pgr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.582573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.582539    5028 generic.go:356] "Generic (PLEG): container finished" podID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerID="7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb" exitCode=0
Oct 30 22:48:12.583035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.582589    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"
Oct 30 22:48:12.583035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.582618    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" event={"ID":"c229eb45-6e20-4d24-bb47-dc47daa3dc0a","Type":"ContainerDied","Data":"7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb"}
Oct 30 22:48:12.583035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.582671    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm" event={"ID":"c229eb45-6e20-4d24-bb47-dc47daa3dc0a","Type":"ContainerDied","Data":"981211fdba9571ca8240efca719bae14022a8eb362f4779a7622127819cdd7a7"}
Oct 30 22:48:12.583035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.582707    5028 scope.go:122] "RemoveContainer" containerID="7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb"
Oct 30 22:48:12.596096 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.596073    5028 scope.go:122] "RemoveContainer" containerID="7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb"
Oct 30 22:48:12.596422 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:12.596403    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb\": container with ID starting with 7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb not found: ID does not exist" containerID="7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb"
Oct 30 22:48:12.596485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.596432    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb"} err="failed to get container status \"7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb\": rpc error: code = NotFound desc = could not find container \"7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb\": container with ID starting with 7ad1e7699ce9bbb45be26d268cf28d24ad82fddfbefadb602426e445ba6eb2cb not found: ID does not exist"
Oct 30 22:48:12.739575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.739546    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:12.752212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.752184    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"]
Oct 30 22:48:12.799477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.799443    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-vdnqm"]
Oct 30 22:48:12.838603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.838530    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca\") pod \"384b2655-148d-493b-9d3a-88affe878640\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") "
Oct 30 22:48:12.838763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.838680    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln\" (UniqueName: \"kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln\") pod \"384b2655-148d-493b-9d3a-88affe878640\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") "
Oct 30 22:48:12.838763 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.838732    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca\") pod \"384b2655-148d-493b-9d3a-88affe878640\" (UID: \"384b2655-148d-493b-9d3a-88affe878640\") "
Oct 30 22:48:12.839111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.839089    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca" pod "384b2655-148d-493b-9d3a-88affe878640" (UID: "384b2655-148d-493b-9d3a-88affe878640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:12.840757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.840732    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca" pod "384b2655-148d-493b-9d3a-88affe878640" (UID: "384b2655-148d-493b-9d3a-88affe878640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:12.840850 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.840779    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln" pod "384b2655-148d-493b-9d3a-88affe878640" (UID: "384b2655-148d-493b-9d3a-88affe878640"). InnerVolumeSpecName "kube-api-access-vgjln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:12.940293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.940261    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjln\" (UniqueName: \"kubernetes.io/projected/384b2655-148d-493b-9d3a-88affe878640-kube-api-access-vgjln\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.940293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.940290    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/384b2655-148d-493b-9d3a-88affe878640-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:12.940293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:12.940301    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/384b2655-148d-493b-9d3a-88affe878640-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:13.291602 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.291565    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" path="/var/lib/kubelet/pods/c229eb45-6e20-4d24-bb47-dc47daa3dc0a/volumes"
Oct 30 22:48:13.593354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.593315    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5" event={"ID":"384b2655-148d-493b-9d3a-88affe878640","Type":"ContainerDied","Data":"2c6ba95d36c95bc6d356d5f702e13df63e38202f34e076e115aec0050ad84283"}
Oct 30 22:48:13.593354 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.593337    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"
Oct 30 22:48:13.593892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.593401    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c6ba95d36c95bc6d356d5f702e13df63e38202f34e076e115aec0050ad84283"
Oct 30 22:48:13.848640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.848571    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:48:13.950051 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950017    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950070    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950102    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950132    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950163    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950192    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950228    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950262    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950314    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token\") pod \"8791da6c-589d-4f64-b334-bc44be58efb7\" (UID: \"8791da6c-589d-4f64-b334-bc44be58efb7\") "
Oct 30 22:48:13.950495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950482    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:13.950745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950505    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:13.950745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950519    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:13.950745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950616    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:13.950923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950886    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8791da6c-589d-4f64-b334-bc44be58efb7-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:13.950923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950906    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:13.950923 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950915    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:13.951086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950929    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:13.951086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.950942    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:13.952795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.952762    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "kube-api-access-g5jnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:13.952795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.952769    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:13.952990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.952803    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:13.953057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:13.953025    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy" pod "8791da6c-589d-4f64-b334-bc44be58efb7" (UID: "8791da6c-589d-4f64-b334-bc44be58efb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:14.052548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.052501    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5jnw\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-kube-api-access-g5jnw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.052548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.052539    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.052548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.052553    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8791da6c-589d-4f64-b334-bc44be58efb7-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.052867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.052566    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8791da6c-589d-4f64-b334-bc44be58efb7-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.052867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.052578    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8791da6c-589d-4f64-b334-bc44be58efb7-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.603878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.603844    5028 generic.go:356] "Generic (PLEG): container finished" podID="8791da6c-589d-4f64-b334-bc44be58efb7" containerID="53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae" exitCode=0
Oct 30 22:48:14.604319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.603916    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"
Oct 30 22:48:14.604319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.603940    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" event={"ID":"8791da6c-589d-4f64-b334-bc44be58efb7","Type":"ContainerDied","Data":"53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae"}
Oct 30 22:48:14.604319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.604011    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv" event={"ID":"8791da6c-589d-4f64-b334-bc44be58efb7","Type":"ContainerDied","Data":"d5b740a7331404d32663e9852521acd5ae697c18aaebd306165dcaf96e91205d"}
Oct 30 22:48:14.604319 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.604048    5028 scope.go:122] "RemoveContainer" containerID="53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae"
Oct 30 22:48:14.614993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.614974    5028 scope.go:122] "RemoveContainer" containerID="06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"
Oct 30 22:48:14.624454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.624434    5028 scope.go:122] "RemoveContainer" containerID="53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae"
Oct 30 22:48:14.624739 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:14.624715    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae\": container with ID starting with 53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae not found: ID does not exist" containerID="53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae"
Oct 30 22:48:14.624843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.624752    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae"} err="failed to get container status \"53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae\": rpc error: code = NotFound desc = could not find container \"53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae\": container with ID starting with 53ddaef8fa1957ed9c4f234573b1ffa2dd27a0564f979fc836704f86adc70cae not found: ID does not exist"
Oct 30 22:48:14.624843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.624778    5028 scope.go:122] "RemoveContainer" containerID="06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"
Oct 30 22:48:14.625060 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:14.625036    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816\": container with ID starting with 06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816 not found: ID does not exist" containerID="06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"
Oct 30 22:48:14.625106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.625071    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816"} err="failed to get container status \"06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816\": rpc error: code = NotFound desc = could not find container \"06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816\": container with ID starting with 06694b714a3618dcb98fad52a70557d4edb48d70d3ddfe085c0a67fa433b9816 not found: ID does not exist"
Oct 30 22:48:14.750146 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.750116    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:14.763210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.763180    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"]
Oct 30 22:48:14.768603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:14.768580    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-afb98c-openshift-default-c8fb6f4c5-qgslv"]
Oct 30 22:48:15.290521 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:15.290486    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" path="/var/lib/kubelet/pods/8791da6c-589d-4f64-b334-bc44be58efb7/volumes"
Oct 30 22:48:15.983264 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:15.983227    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:19.034828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:19.034797    5028 scope.go:122] "RemoveContainer" containerID="f8747f2e71ed0b9df584e453e8b421e50e62c7d2b809b206dc30120b35ef24ed"
Oct 30 22:48:19.044812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:19.044794    5028 scope.go:122] "RemoveContainer" containerID="0aa4c29139345efbe2c4e33d1be8e5cab21f56ad11cc36574de35b86465d2f6a"
Oct 30 22:48:19.056773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:19.056747    5028 scope.go:122] "RemoveContainer" containerID="f186bfe394d7db5f2b3302fc63b13b6852bb6cc2360bb7a88a6f126d3c97a1d7"
Oct 30 22:48:19.066149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:19.066128    5028 scope.go:122] "RemoveContainer" containerID="4e5c9ed3e50d7f9f23eb428eabb10e09fa80380faeefcdb762bba23569652fa2"
Oct 30 22:48:19.250752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:19.250661    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:20.800239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:20.800193    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"]
Oct 30 22:48:20.900784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:20.900741    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"]
Oct 30 22:48:21.157884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:21.157800    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:21.324456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:21.324424    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:23.660821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:23.660789    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:25.800968 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:25.800877    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh" podUID="68d431a0-67bb-469a-ae0c-b672ae71f7fd" containerName="payload-processing" containerID="cri-o://fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484" gracePeriod=25
Oct 30 22:48:25.901653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:25.901576    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh" podUID="6a0f919c-bad2-48f0-8b52-a70234598d71" containerName="payload-pre-processing" containerID="cri-o://e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa" gracePeriod=25
Oct 30 22:48:26.057703 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.057616    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:48:26.175977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.175949    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w\" (UniqueName: \"kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w\") pod \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") "
Oct 30 22:48:26.176156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.176020    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume\") pod \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\" (UID: \"68d431a0-67bb-469a-ae0c-b672ae71f7fd\") "
Oct 30 22:48:26.176442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.176417    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume" pod "68d431a0-67bb-469a-ae0c-b672ae71f7fd" (UID: "68d431a0-67bb-469a-ae0c-b672ae71f7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:26.177216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.177193    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:48:26.178210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.178186    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w" pod "68d431a0-67bb-469a-ae0c-b672ae71f7fd" (UID: "68d431a0-67bb-469a-ae0c-b672ae71f7fd"). InnerVolumeSpecName "kube-api-access-h9z2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:26.276630 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.276594    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume\") pod \"6a0f919c-bad2-48f0-8b52-a70234598d71\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") "
Oct 30 22:48:26.276809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.276679    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf\" (UniqueName: \"kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf\") pod \"6a0f919c-bad2-48f0-8b52-a70234598d71\" (UID: \"6a0f919c-bad2-48f0-8b52-a70234598d71\") "
Oct 30 22:48:26.276939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.276925    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9z2w\" (UniqueName: \"kubernetes.io/projected/68d431a0-67bb-469a-ae0c-b672ae71f7fd-kube-api-access-h9z2w\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:26.276996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.276941    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d431a0-67bb-469a-ae0c-b672ae71f7fd-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:26.277036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.277000    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume" pod "6a0f919c-bad2-48f0-8b52-a70234598d71" (UID: "6a0f919c-bad2-48f0-8b52-a70234598d71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:26.278724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.278705    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf" pod "6a0f919c-bad2-48f0-8b52-a70234598d71" (UID: "6a0f919c-bad2-48f0-8b52-a70234598d71"). InnerVolumeSpecName "kube-api-access-5mtwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:26.378381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.378286    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0f919c-bad2-48f0-8b52-a70234598d71-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:26.378381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.378326    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtwf\" (UniqueName: \"kubernetes.io/projected/6a0f919c-bad2-48f0-8b52-a70234598d71-kube-api-access-5mtwf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:26.695444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.695352    5028 generic.go:356] "Generic (PLEG): container finished" podID="68d431a0-67bb-469a-ae0c-b672ae71f7fd" containerID="fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484" exitCode=0
Oct 30 22:48:26.695444 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.695401    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"
Oct 30 22:48:26.695712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.695437    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh" event={"ID":"68d431a0-67bb-469a-ae0c-b672ae71f7fd","Type":"ContainerDied","Data":"fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484"}
Oct 30 22:48:26.695712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.695502    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh" event={"ID":"68d431a0-67bb-469a-ae0c-b672ae71f7fd","Type":"ContainerDied","Data":"7df743739363e86a06ec9499f88a629763d5c50ef3435397bbc2f5e59e973591"}
Oct 30 22:48:26.695712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.695531    5028 scope.go:122] "RemoveContainer" containerID="fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484"
Oct 30 22:48:26.696952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.696930    5028 generic.go:356] "Generic (PLEG): container finished" podID="6a0f919c-bad2-48f0-8b52-a70234598d71" containerID="e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa" exitCode=0
Oct 30 22:48:26.697083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.696981    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"
Oct 30 22:48:26.697083 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.696985    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh" event={"ID":"6a0f919c-bad2-48f0-8b52-a70234598d71","Type":"ContainerDied","Data":"e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa"}
Oct 30 22:48:26.697237 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.697108    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh" event={"ID":"6a0f919c-bad2-48f0-8b52-a70234598d71","Type":"ContainerDied","Data":"5724c8b4902a8eba7b8f15b78c44fc1363c918e063b8f6254cc4a2b7daea5930"}
Oct 30 22:48:26.706217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.706194    5028 scope.go:122] "RemoveContainer" containerID="fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484"
Oct 30 22:48:26.706594 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:26.706562    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484\": container with ID starting with fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484 not found: ID does not exist" containerID="fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484"
Oct 30 22:48:26.706739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.706605    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484"} err="failed to get container status \"fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484\": rpc error: code = NotFound desc = could not find container \"fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484\": container with ID starting with fd83f9b0096430fd83a4c727275f30a2092440c5d6e1631830abcf1485594484 not found: ID does not exist"
Oct 30 22:48:26.706739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.706632    5028 scope.go:122] "RemoveContainer" containerID="e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa"
Oct 30 22:48:26.716466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.716448    5028 scope.go:122] "RemoveContainer" containerID="e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa"
Oct 30 22:48:26.716780 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:26.716761    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa\": container with ID starting with e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa not found: ID does not exist" containerID="e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa"
Oct 30 22:48:26.716828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.716789    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa"} err="failed to get container status \"e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa\": rpc error: code = NotFound desc = could not find container \"e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa\": container with ID starting with e251f33f9c3d8f1493f3e0c17edf0e4afa9ea5ae6792b19984fe92a1a229bdfa not found: ID does not exist"
Oct 30 22:48:26.857617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.857579    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"]
Oct 30 22:48:26.862180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.862153    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-926fc6-7776b58857-8d7xh"]
Oct 30 22:48:26.965635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.965598    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"]
Oct 30 22:48:26.969150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:26.969128    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-926fc6-5c557ffccf-27hbh"]
Oct 30 22:48:27.291163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:27.291068    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d431a0-67bb-469a-ae0c-b672ae71f7fd" path="/var/lib/kubelet/pods/68d431a0-67bb-469a-ae0c-b672ae71f7fd/volumes"
Oct 30 22:48:27.291457 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:27.291442    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0f919c-bad2-48f0-8b52-a70234598d71" path="/var/lib/kubelet/pods/6a0f919c-bad2-48f0-8b52-a70234598d71/volumes"
Oct 30 22:48:28.223983 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:28.223949    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:28.436025 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:28.435982    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:48:32.752876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:32.752840    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:34.839494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:34.839454    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:34.998045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:34.998002    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:36.743633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:36.743596    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:38.229606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:38.229571    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:38.473125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:38.473083    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:38.773518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:38.773480    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"]
Oct 30 22:48:38.775730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:38.775654    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:38.820315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:38.820282    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-926fc6-wz7b5"]
Oct 30 22:48:39.221594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:39.221561    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:39.290338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:39.290296    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384b2655-148d-493b-9d3a-88affe878640" path="/var/lib/kubelet/pods/384b2655-148d-493b-9d3a-88affe878640/volumes"
Oct 30 22:48:43.236006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:43.235966    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:45.656864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:45.656830    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:45.812211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:45.812174    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:46.030002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.029968    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:46.917060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917019    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"]
Oct 30 22:48:46.917503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917487    5028 cpu_manager.go:414] "containerMap: removing container" podUID="68d431a0-67bb-469a-ae0c-b672ae71f7fd" containerName="payload-processing"
Oct 30 22:48:46.917549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917504    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d431a0-67bb-469a-ae0c-b672ae71f7fd" containerName="payload-processing"
Oct 30 22:48:46.917549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917522    5028 cpu_manager.go:414] "containerMap: removing container" podUID="6a0f919c-bad2-48f0-8b52-a70234598d71" containerName="payload-pre-processing"
Oct 30 22:48:46.917549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917530    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0f919c-bad2-48f0-8b52-a70234598d71" containerName="payload-pre-processing"
Oct 30 22:48:46.917549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917542    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy"
Oct 30 22:48:46.917549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917548    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917562    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917568    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917590    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917595    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917612    5028 cpu_manager.go:414] "containerMap: removing container" podUID="384b2655-148d-493b-9d3a-88affe878640" containerName="revoke-keys"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917619    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384b2655-148d-493b-9d3a-88affe878640" containerName="revoke-keys"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917726    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0f919c-bad2-48f0-8b52-a70234598d71" containerName="payload-pre-processing"
Oct 30 22:48:46.917738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917736    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy"
Oct 30 22:48:46.917989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917744    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8791da6c-589d-4f64-b334-bc44be58efb7" containerName="istio-proxy"
Oct 30 22:48:46.917989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917754    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c229eb45-6e20-4d24-bb47-dc47daa3dc0a" containerName="istio-proxy"
Oct 30 22:48:46.917989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917760    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="384b2655-148d-493b-9d3a-88affe878640" containerName="revoke-keys"
Oct 30 22:48:46.917989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.917770    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d431a0-67bb-469a-ae0c-b672ae71f7fd" containerName="payload-processing"
Oct 30 22:48:46.922343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.922321    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:46.965857 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:46.965826    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"]
Oct 30 22:48:47.009187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.009152    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:47.009187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.009166    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:47.009187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.009184    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:47.019663 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.019643    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:48:47.085622 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.085582    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h79j\" (UniqueName: \"kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.085799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.085635    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.085873 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.085851    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.187022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.186925    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.187022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.187019    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.187280 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.187137    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h79j\" (UniqueName: \"kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.187668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.187645    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.189374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.189352    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.246422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.246389    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h79j\" (UniqueName: \"kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.536271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.536238    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:47.909108 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.909078    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"]
Oct 30 22:48:47.909770 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:48:47.909738    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6e8a6c_9629_4dd1_be17_17cec8aa0a7e.slice/crio-a566d6e6f179806417db91966a7f4444950aa1efec96999fbc3216fe709adbeb WatchSource:0}: Error finding container a566d6e6f179806417db91966a7f4444950aa1efec96999fbc3216fe709adbeb: Status 404 returned error can't find the container with id a566d6e6f179806417db91966a7f4444950aa1efec96999fbc3216fe709adbeb
Oct 30 22:48:47.964723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:47.964677    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:48.845211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:48.845178    5028 generic.go:356] "Generic (PLEG): container finished" podID="fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" containerID="8cca0ce12d410e45237d96d6ed51b97e2d67119552de8baff415d06445a83fe2" exitCode=0
Oct 30 22:48:48.845357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:48.845210    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9" event={"ID":"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e","Type":"ContainerDied","Data":"8cca0ce12d410e45237d96d6ed51b97e2d67119552de8baff415d06445a83fe2"}
Oct 30 22:48:48.845357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:48.845265    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9" event={"ID":"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e","Type":"ContainerStarted","Data":"a566d6e6f179806417db91966a7f4444950aa1efec96999fbc3216fe709adbeb"}
Oct 30 22:48:49.988600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:49.988571    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:50.115485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.115400    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca\") pod \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") "
Oct 30 22:48:50.115671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.115527    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca\") pod \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") "
Oct 30 22:48:50.115770 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.115662    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j\" (UniqueName: \"kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j\") pod \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\" (UID: \"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e\") "
Oct 30 22:48:50.115887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.115864    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca" pod "fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" (UID: "fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:50.116084 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.116067    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:50.117615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.117592    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca" pod "fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" (UID: "fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:50.117890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.117866    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j" pod "fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" (UID: "fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e"). InnerVolumeSpecName "kube-api-access-7h79j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:50.217467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.217435    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:50.217467 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.217465    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h79j\" (UniqueName: \"kubernetes.io/projected/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e-kube-api-access-7h79j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:50.859771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.859735    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9" event={"ID":"fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e","Type":"ContainerDied","Data":"a566d6e6f179806417db91966a7f4444950aa1efec96999fbc3216fe709adbeb"}
Oct 30 22:48:50.859771 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.859757    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"
Oct 30 22:48:50.859995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:50.859788    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a566d6e6f179806417db91966a7f4444950aa1efec96999fbc3216fe709adbeb"
Oct 30 22:48:52.744067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:52.744035    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:57.247389 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:57.247352    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.351576 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:58.351541    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"]
Oct 30 22:48:58.436002 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:48:58.435963    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:48:58.452153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:58.452122    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"]
Oct 30 22:48:58.722591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:58.722541    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.894820 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:48:58.894781    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:01.767314 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:01.767228    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:03.352043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.351975    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc" podUID="e3f74408-b90d-4335-ac65-afe19b9bf935" containerName="payload-processing" containerID="cri-o://2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889" gracePeriod=25
Oct 30 22:49:03.452733 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.452647    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" podUID="fe445dba-eacb-43e9-9745-9c569c194037" containerName="payload-pre-processing" containerID="cri-o://2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f" gracePeriod=25
Oct 30 22:49:03.614402 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:49:03.614374    5028 logging.go:55] [core] [Channel #11507 SubChannel #11508]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.186:50052", ServerName: "10.132.0.186:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.186:50052: connect: connection refused"
Oct 30 22:49:03.635280 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.635251    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:49:03.714541 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.714517    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:49:03.742949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.742913    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume\") pod \"e3f74408-b90d-4335-ac65-afe19b9bf935\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") "
Oct 30 22:49:03.743117 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.743042    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8\" (UniqueName: \"kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8\") pod \"e3f74408-b90d-4335-ac65-afe19b9bf935\" (UID: \"e3f74408-b90d-4335-ac65-afe19b9bf935\") "
Oct 30 22:49:03.743327 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.743296    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume" pod "e3f74408-b90d-4335-ac65-afe19b9bf935" (UID: "e3f74408-b90d-4335-ac65-afe19b9bf935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:03.743443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.743421    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3f74408-b90d-4335-ac65-afe19b9bf935-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.745187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.745164    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8" pod "e3f74408-b90d-4335-ac65-afe19b9bf935" (UID: "e3f74408-b90d-4335-ac65-afe19b9bf935"). InnerVolumeSpecName "kube-api-access-48rm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:03.844516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.844478    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg\" (UniqueName: \"kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg\") pod \"fe445dba-eacb-43e9-9745-9c569c194037\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") "
Oct 30 22:49:03.844724 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.844542    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume\") pod \"fe445dba-eacb-43e9-9745-9c569c194037\" (UID: \"fe445dba-eacb-43e9-9745-9c569c194037\") "
Oct 30 22:49:03.844843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.844830    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rm8\" (UniqueName: \"kubernetes.io/projected/e3f74408-b90d-4335-ac65-afe19b9bf935-kube-api-access-48rm8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.844947 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.844924    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume" pod "fe445dba-eacb-43e9-9745-9c569c194037" (UID: "fe445dba-eacb-43e9-9745-9c569c194037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:03.846611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.846590    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg" pod "fe445dba-eacb-43e9-9745-9c569c194037" (UID: "fe445dba-eacb-43e9-9745-9c569c194037"). InnerVolumeSpecName "kube-api-access-gxshg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:03.945439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.945328    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxshg\" (UniqueName: \"kubernetes.io/projected/fe445dba-eacb-43e9-9745-9c569c194037-kube-api-access-gxshg\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.945439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.945366    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe445dba-eacb-43e9-9745-9c569c194037-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.952613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.952580    5028 generic.go:356] "Generic (PLEG): container finished" podID="fe445dba-eacb-43e9-9745-9c569c194037" containerID="2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f" exitCode=0
Oct 30 22:49:03.952795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.952641    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"
Oct 30 22:49:03.952795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.952669    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" event={"ID":"fe445dba-eacb-43e9-9745-9c569c194037","Type":"ContainerDied","Data":"2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f"}
Oct 30 22:49:03.952795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.952748    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" event={"ID":"fe445dba-eacb-43e9-9745-9c569c194037","Type":"ContainerDied","Data":"f6df51895eec6a45fe140a19707c387ea8fe3760135b750d8a3603942496d476"}
Oct 30 22:49:03.952795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.952780    5028 scope.go:122] "RemoveContainer" containerID="2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f"
Oct 30 22:49:03.954268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.954240    5028 generic.go:356] "Generic (PLEG): container finished" podID="e3f74408-b90d-4335-ac65-afe19b9bf935" containerID="2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889" exitCode=0
Oct 30 22:49:03.954380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.954279    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc" event={"ID":"e3f74408-b90d-4335-ac65-afe19b9bf935","Type":"ContainerDied","Data":"2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889"}
Oct 30 22:49:03.954380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.954291    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"
Oct 30 22:49:03.954380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.954319    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc" event={"ID":"e3f74408-b90d-4335-ac65-afe19b9bf935","Type":"ContainerDied","Data":"74f332a4c55a0c89f72493ee26472cc899a512187de5092995592dc5b74e6041"}
Oct 30 22:49:03.964202 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.964175    5028 scope.go:122] "RemoveContainer" containerID="2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f"
Oct 30 22:49:03.964518 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:03.964498    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f\": container with ID starting with 2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f not found: ID does not exist" containerID="2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f"
Oct 30 22:49:03.964618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.964526    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f"} err="failed to get container status \"2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f\": rpc error: code = NotFound desc = could not find container \"2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f\": container with ID starting with 2f49642882f486321d49afcbd2e93b6b764dafec710504ab0c11289fba65223f not found: ID does not exist"
Oct 30 22:49:03.964618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.964543    5028 scope.go:122] "RemoveContainer" containerID="2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889"
Oct 30 22:49:03.975370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.975344    5028 scope.go:122] "RemoveContainer" containerID="2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889"
Oct 30 22:49:03.975736 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:03.975681    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889\": container with ID starting with 2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889 not found: ID does not exist" containerID="2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889"
Oct 30 22:49:03.975806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:03.975746    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889"} err="failed to get container status \"2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889\": rpc error: code = NotFound desc = could not find container \"2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889\": container with ID starting with 2c48b850de30074746cd017ba030b8910fafe758373078770eea3f40994e0889 not found: ID does not exist"
Oct 30 22:49:04.110617 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:04.110582    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"]
Oct 30 22:49:04.115250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:04.115219    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5c2f68-5dbf8fbf6c-h6pqc"]
Oct 30 22:49:04.220219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:04.220189    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"]
Oct 30 22:49:04.224943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:04.224917    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57"]
Oct 30 22:49:04.614752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:04.614656    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5c2f68-665485d7f5-rxz57" podUID="fe445dba-eacb-43e9-9745-9c569c194037" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.186:50052\" within 1s: context deadline exceeded"
Oct 30 22:49:05.290046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:05.290011    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f74408-b90d-4335-ac65-afe19b9bf935" path="/var/lib/kubelet/pods/e3f74408-b90d-4335-ac65-afe19b9bf935/volumes"
Oct 30 22:49:05.290390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:05.290377    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe445dba-eacb-43e9-9745-9c569c194037" path="/var/lib/kubelet/pods/fe445dba-eacb-43e9-9745-9c569c194037/volumes"
Oct 30 22:49:05.770977 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:05.770946    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:07.242493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:07.242447    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:07.443033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:07.442980    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:07.594006 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:07.593917    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:07.727231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:07.727193    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"]
Oct 30 22:49:07.760418 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:07.760382    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5c2f68-46rr9"]
Oct 30 22:49:08.121698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:08.121646    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:09.292311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:09.292279    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" path="/var/lib/kubelet/pods/fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e/volumes"
Oct 30 22:49:12.179356 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:12.179321    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:12.341164 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:12.341126    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:14.442706 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:14.442658    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:14.645037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:14.644993    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:14.790517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:14.790475    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"]
Oct 30 22:49:14.790801 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:14.790776    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy" containerID="cri-o://b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da" gracePeriod=30
Oct 30 22:49:14.815520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:14.815490    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:15.294665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.294632    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:15.802130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802090    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"]
Oct 30 22:49:15.802531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802517    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" containerName="revoke-keys"
Oct 30 22:49:15.802531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802527    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" containerName="revoke-keys"
Oct 30 22:49:15.802613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802555    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e3f74408-b90d-4335-ac65-afe19b9bf935" containerName="payload-processing"
Oct 30 22:49:15.802613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802561    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f74408-b90d-4335-ac65-afe19b9bf935" containerName="payload-processing"
Oct 30 22:49:15.802613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802570    5028 cpu_manager.go:414] "containerMap: removing container" podUID="fe445dba-eacb-43e9-9745-9c569c194037" containerName="payload-pre-processing"
Oct 30 22:49:15.802613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802575    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe445dba-eacb-43e9-9745-9c569c194037" containerName="payload-pre-processing"
Oct 30 22:49:15.802776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802650    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe445dba-eacb-43e9-9745-9c569c194037" containerName="payload-pre-processing"
Oct 30 22:49:15.802776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802662    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6e8a6c-9629-4dd1-be17-17cec8aa0a7e" containerName="revoke-keys"
Oct 30 22:49:15.802776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.802673    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f74408-b90d-4335-ac65-afe19b9bf935" containerName="payload-processing"
Oct 30 22:49:15.806191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.806174    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.872970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.872933    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qk2\" (UniqueName: \"kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.873174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.873089    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.873174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.873118    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.905601 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.905564    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:15.905863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.905598    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:49:15.905863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.905715    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:15.906031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.905747    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:15.908919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.908895    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"]
Oct 30 22:49:15.974636 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.974595    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.974894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.974740    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qk2\" (UniqueName: \"kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.975003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.974890    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.975479 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.975458    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:15.977307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:15.977289    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:16.033426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:16.033391    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qk2\" (UniqueName: \"kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2\") pod \"maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:16.119182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:16.119088    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:16.320394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:16.320360    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:16.488371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:16.488339    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"]
Oct 30 22:49:16.489064 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:49:16.489035    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057f52f1_4d31_401b_8112_20bfd8643a33.slice/crio-06ec29d633edcc053120ef4e37745461baf89e8aa5107f7500b44cfa0027adaf WatchSource:0}: Error finding container 06ec29d633edcc053120ef4e37745461baf89e8aa5107f7500b44cfa0027adaf: Status 404 returned error can't find the container with id 06ec29d633edcc053120ef4e37745461baf89e8aa5107f7500b44cfa0027adaf
Oct 30 22:49:17.060459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:17.060361    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2" event={"ID":"057f52f1-4d31-401b-8112-20bfd8643a33","Type":"ContainerStarted","Data":"6daa61f174776a1111808acfbde2da65db6ba2345048aa60dbd22dfc7c0ba4c0"}
Oct 30 22:49:17.060459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:17.060433    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2" event={"ID":"057f52f1-4d31-401b-8112-20bfd8643a33","Type":"ContainerStarted","Data":"06ec29d633edcc053120ef4e37745461baf89e8aa5107f7500b44cfa0027adaf"}
Oct 30 22:49:17.179595 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:17.179542    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2" podStartSLOduration=2.179526084 podStartE2EDuration="2.179526084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:17.177117172 +0000 UTC m=+2888.026630146" watchObservedRunningTime="2026-10-30 22:49:17.179526084 +0000 UTC m=+2888.029039056"
Oct 30 22:49:17.182603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:17.182574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:18.068038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:18.067993    5028 generic.go:356] "Generic (PLEG): container finished" podID="057f52f1-4d31-401b-8112-20bfd8643a33" containerID="6daa61f174776a1111808acfbde2da65db6ba2345048aa60dbd22dfc7c0ba4c0" exitCode=0
Oct 30 22:49:18.068514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:18.068076    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2" event={"ID":"057f52f1-4d31-401b-8112-20bfd8643a33","Type":"ContainerDied","Data":"6daa61f174776a1111808acfbde2da65db6ba2345048aa60dbd22dfc7c0ba4c0"}
Oct 30 22:49:19.154903 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.154854    5028 scope.go:122] "RemoveContainer" containerID="fe4a41b5c9153e005c03cac747dba67868cbe9489e1f08b8bbe95f10ae3b62d7"
Oct 30 22:49:19.231589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.231550    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:19.305183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.305149    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2\" (UniqueName: \"kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2\") pod \"057f52f1-4d31-401b-8112-20bfd8643a33\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") "
Oct 30 22:49:19.305377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.305207    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca\") pod \"057f52f1-4d31-401b-8112-20bfd8643a33\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") "
Oct 30 22:49:19.305377 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.305276    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca\") pod \"057f52f1-4d31-401b-8112-20bfd8643a33\" (UID: \"057f52f1-4d31-401b-8112-20bfd8643a33\") "
Oct 30 22:49:19.305534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.305512    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca" pod "057f52f1-4d31-401b-8112-20bfd8643a33" (UID: "057f52f1-4d31-401b-8112-20bfd8643a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:19.305655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.305639    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.307321 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.307299    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2" pod "057f52f1-4d31-401b-8112-20bfd8643a33" (UID: "057f52f1-4d31-401b-8112-20bfd8643a33"). InnerVolumeSpecName "kube-api-access-x4qk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:19.307391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.307316    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca" pod "057f52f1-4d31-401b-8112-20bfd8643a33" (UID: "057f52f1-4d31-401b-8112-20bfd8643a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:19.407114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.407032    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4qk2\" (UniqueName: \"kubernetes.io/projected/057f52f1-4d31-401b-8112-20bfd8643a33-kube-api-access-x4qk2\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.407114 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:19.407061    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057f52f1-4d31-401b-8112-20bfd8643a33-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.038781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.038753    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:49:20.086157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.086068    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"
Oct 30 22:49:20.086157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.086089    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2" event={"ID":"057f52f1-4d31-401b-8112-20bfd8643a33","Type":"ContainerDied","Data":"06ec29d633edcc053120ef4e37745461baf89e8aa5107f7500b44cfa0027adaf"}
Oct 30 22:49:20.086157 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.086156    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ec29d633edcc053120ef4e37745461baf89e8aa5107f7500b44cfa0027adaf"
Oct 30 22:49:20.088161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.088129    5028 generic.go:356] "Generic (PLEG): container finished" podID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerID="b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da" exitCode=0
Oct 30 22:49:20.088320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.088185    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"
Oct 30 22:49:20.088320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.088208    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" event={"ID":"b1b27184-4cbd-46d1-ade8-c061d5da46a4","Type":"ContainerDied","Data":"b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da"}
Oct 30 22:49:20.088320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.088259    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77" event={"ID":"b1b27184-4cbd-46d1-ade8-c061d5da46a4","Type":"ContainerDied","Data":"691be7ac2d07cd3213bf6e53a0dac404ede0ba1c2f85822354fe0b76dff38ac1"}
Oct 30 22:49:20.088320 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.088288    5028 scope.go:122] "RemoveContainer" containerID="b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da"
Oct 30 22:49:20.098738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.098723    5028 scope.go:122] "RemoveContainer" containerID="4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"
Oct 30 22:49:20.107788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.107771    5028 scope.go:122] "RemoveContainer" containerID="b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da"
Oct 30 22:49:20.108026 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:20.107998    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da\": container with ID starting with b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da not found: ID does not exist" containerID="b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da"
Oct 30 22:49:20.108086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.108031    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da"} err="failed to get container status \"b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da\": rpc error: code = NotFound desc = could not find container \"b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da\": container with ID starting with b9a9da1832c3ad6859b192a6eb6ddcd8054837deed291bf9b9aa566ecbfc49da not found: ID does not exist"
Oct 30 22:49:20.108086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.108049    5028 scope.go:122] "RemoveContainer" containerID="4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"
Oct 30 22:49:20.108276 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:20.108256    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9\": container with ID starting with 4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9 not found: ID does not exist" containerID="4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"
Oct 30 22:49:20.108318 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.108283    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9"} err="failed to get container status \"4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9\": rpc error: code = NotFound desc = could not find container \"4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9\": container with ID starting with 4708f2886d3b2f521421c2008d5db60d6932c4198aaa1f6189aead4ce8ddefc9 not found: ID does not exist"
Oct 30 22:49:20.113565 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113548    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.113646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113596    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.113646 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113630    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.113939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113656    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.113939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113807    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.113939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113863    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.113939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113866    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.113939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113911    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.114251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113969    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.114251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.113977    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.114251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114008    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert\") pod \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\" (UID: \"b1b27184-4cbd-46d1-ade8-c061d5da46a4\") "
Oct 30 22:49:20.114251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114037    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.114251 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114100    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.114504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114395    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.114504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114418    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.114504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114434    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.114504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114448    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.114504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.114458    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:20.115988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.115962    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:20.115988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.115973    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:20.116210 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.116194    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "kube-api-access-4zp8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:20.116261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.116243    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy" pod "b1b27184-4cbd-46d1-ade8-c061d5da46a4" (UID: "b1b27184-4cbd-46d1-ade8-c061d5da46a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:20.215989 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.215958    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.216414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.215991    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.216414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.216005    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.216414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.216018    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b27184-4cbd-46d1-ade8-c061d5da46a4-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.216414 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.216038    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zp8j\" (UniqueName: \"kubernetes.io/projected/b1b27184-4cbd-46d1-ade8-c061d5da46a4-kube-api-access-4zp8j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.462661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.462630    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:20.508514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.508471    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"]
Oct 30 22:49:20.513649 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:20.513617    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5c2f68-openshift-default-548b8f465c-spk77"]
Oct 30 22:49:21.290632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:21.290596    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" path="/var/lib/kubelet/pods/b1b27184-4cbd-46d1-ade8-c061d5da46a4/volumes"
Oct 30 22:49:22.039554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:22.039523    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:26.667661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:26.667624    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:27.257821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:27.257783    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"]
Oct 30 22:49:27.359034 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:27.358998    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"]
Oct 30 22:49:27.599753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:27.599638    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:27.757973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:27.757937    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:28.436385 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:28.436349    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:49:31.172349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:31.172305    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:32.258299 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:32.258259    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerName="payload-processing" containerID="cri-o://fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54" gracePeriod=25
Oct 30 22:49:32.359813 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:32.359756    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" podUID="919be555-ac42-475b-932e-fedb50ebea01" containerName="payload-pre-processing" containerID="cri-o://3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924" gracePeriod=25
Oct 30 22:49:35.307811 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.307777    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"]
Oct 30 22:49:35.308234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308221    5028 cpu_manager.go:414] "containerMap: removing container" podUID="057f52f1-4d31-401b-8112-20bfd8643a33" containerName="revoke-keys"
Oct 30 22:49:35.308277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308235    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057f52f1-4d31-401b-8112-20bfd8643a33" containerName="revoke-keys"
Oct 30 22:49:35.308277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308247    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy"
Oct 30 22:49:35.308277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308253    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy"
Oct 30 22:49:35.308381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308332    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="057f52f1-4d31-401b-8112-20bfd8643a33" containerName="revoke-keys"
Oct 30 22:49:35.308381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308339    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy"
Oct 30 22:49:35.308381 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.308349    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy"
Oct 30 22:49:35.313193 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.313164    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.366446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.366413    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"]
Oct 30 22:49:35.410333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.410303    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:35.410333 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.410303    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:35.410542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.410362    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:49:35.410542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.410410    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:35.468637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.468600    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.468637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.468644    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.468891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.468833    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2k6f\" (UniqueName: \"kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.569679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.569590    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2k6f\" (UniqueName: \"kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.569679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.569634    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.569679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.569657    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.570293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.570265    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.572025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.572003    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.628451 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.628413    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2k6f\" (UniqueName: \"kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f\") pod \"maas-api-revoke-sub-1ad84f8a-7m2mn\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:35.927113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:35.927039    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:36.298872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:36.298829    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"]
Oct 30 22:49:36.299943 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:49:36.299915    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dd0114_0c8c_4bab_83fe_e6df2c18175c.slice/crio-5d59ccd81f59aaef8ed9a69f16117129f7789ee93085489da73e610de097c2ab WatchSource:0}: Error finding container 5d59ccd81f59aaef8ed9a69f16117129f7789ee93085489da73e610de097c2ab: Status 404 returned error can't find the container with id 5d59ccd81f59aaef8ed9a69f16117129f7789ee93085489da73e610de097c2ab
Oct 30 22:49:37.211295 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:37.211261    5028 generic.go:356] "Generic (PLEG): container finished" podID="67dd0114-0c8c-4bab-83fe-e6df2c18175c" containerID="cdc45cf73c4ba1f7f4ae62093aa0e9ecd44ac16f827deb3d9baa4029c66aa49e" exitCode=0
Oct 30 22:49:37.211726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:37.211354    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn" event={"ID":"67dd0114-0c8c-4bab-83fe-e6df2c18175c","Type":"ContainerDied","Data":"cdc45cf73c4ba1f7f4ae62093aa0e9ecd44ac16f827deb3d9baa4029c66aa49e"}
Oct 30 22:49:37.211726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:37.211418    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn" event={"ID":"67dd0114-0c8c-4bab-83fe-e6df2c18175c","Type":"ContainerStarted","Data":"5d59ccd81f59aaef8ed9a69f16117129f7789ee93085489da73e610de097c2ab"}
Oct 30 22:49:37.397481 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:37.397398    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" podUID="919be555-ac42-475b-932e-fedb50ebea01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:38.349787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.349757    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:38.406035 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.405998    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:38.498922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.498845    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca\") pod \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") "
Oct 30 22:49:38.499063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.498920    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca\") pod \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") "
Oct 30 22:49:38.499063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.498985    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f\" (UniqueName: \"kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f\") pod \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\" (UID: \"67dd0114-0c8c-4bab-83fe-e6df2c18175c\") "
Oct 30 22:49:38.499336 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.499311    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca" pod "67dd0114-0c8c-4bab-83fe-e6df2c18175c" (UID: "67dd0114-0c8c-4bab-83fe-e6df2c18175c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:38.501182 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.501154    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca" pod "67dd0114-0c8c-4bab-83fe-e6df2c18175c" (UID: "67dd0114-0c8c-4bab-83fe-e6df2c18175c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:38.501316 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.501262    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f" pod "67dd0114-0c8c-4bab-83fe-e6df2c18175c" (UID: "67dd0114-0c8c-4bab-83fe-e6df2c18175c"). InnerVolumeSpecName "kube-api-access-m2k6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:38.600865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.600832    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:38.600865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.600859    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dd0114-0c8c-4bab-83fe-e6df2c18175c-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:38.600865 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:38.600870    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2k6f\" (UniqueName: \"kubernetes.io/projected/67dd0114-0c8c-4bab-83fe-e6df2c18175c-kube-api-access-m2k6f\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:39.226679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:39.226649    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"
Oct 30 22:49:39.226679 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:39.226695    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn" event={"ID":"67dd0114-0c8c-4bab-83fe-e6df2c18175c","Type":"ContainerDied","Data":"5d59ccd81f59aaef8ed9a69f16117129f7789ee93085489da73e610de097c2ab"}
Oct 30 22:49:39.226935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:39.226746    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d59ccd81f59aaef8ed9a69f16117129f7789ee93085489da73e610de097c2ab"
Oct 30 22:49:47.397012 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:47.396967    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" podUID="919be555-ac42-475b-932e-fedb50ebea01" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:48.406243 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:48.406206    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:50.738616 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:50.738524    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.134405 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.134367    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.405818 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:52.405738    5028 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919be555_ac42_475b_932e_fedb50ebea01.slice/crio-conmon-3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:49:52.407240 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.407205    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:52.531949 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.531924    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:49:52.617364 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.617339    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:49:52.632600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.632574    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"]
Oct 30 22:49:52.635468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.635440    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eb4aa1-qg8f2"]
Oct 30 22:49:52.647867 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.647840    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj\" (UniqueName: \"kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj\") pod \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") "
Oct 30 22:49:52.648044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.647884    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume\") pod \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\" (UID: \"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a\") "
Oct 30 22:49:52.648044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.647916    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8\") pod \"919be555-ac42-475b-932e-fedb50ebea01\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") "
Oct 30 22:49:52.648044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.648000    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume\") pod \"919be555-ac42-475b-932e-fedb50ebea01\" (UID: \"919be555-ac42-475b-932e-fedb50ebea01\") "
Oct 30 22:49:52.648461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.648433    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume" pod "5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" (UID: "5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:52.648607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.648478    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume" pod "919be555-ac42-475b-932e-fedb50ebea01" (UID: "919be555-ac42-475b-932e-fedb50ebea01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:52.650643 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.650619    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj" pod "5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" (UID: "5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a"). InnerVolumeSpecName "kube-api-access-r6tdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:52.650790 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.650637    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8" pod "919be555-ac42-475b-932e-fedb50ebea01" (UID: "919be555-ac42-475b-932e-fedb50ebea01"). InnerVolumeSpecName "kube-api-access-tcqh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:52.749118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.749080    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6tdj\" (UniqueName: \"kubernetes.io/projected/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-kube-api-access-r6tdj\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:52.749118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.749112    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:52.749118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.749122    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/919be555-ac42-475b-932e-fedb50ebea01-kube-api-access-tcqh8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:52.749371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:52.749131    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919be555-ac42-475b-932e-fedb50ebea01-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:53.042355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.042275    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:53.290007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.289975    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057f52f1-4d31-401b-8112-20bfd8643a33" path="/var/lib/kubelet/pods/057f52f1-4d31-401b-8112-20bfd8643a33/volumes"
Oct 30 22:49:53.351287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.351256    5028 generic.go:356] "Generic (PLEG): container finished" podID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerID="fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54" exitCode=0
Oct 30 22:49:53.351475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.351307    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"
Oct 30 22:49:53.351475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.351336    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" event={"ID":"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a","Type":"ContainerDied","Data":"fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54"}
Oct 30 22:49:53.351475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.351385    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2" event={"ID":"5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a","Type":"ContainerDied","Data":"8de1d66b3e8c159a85ec2c2004264eebf0c59974b1cb057c8429295476c72820"}
Oct 30 22:49:53.351475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.351412    5028 scope.go:122] "RemoveContainer" containerID="fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54"
Oct 30 22:49:53.352877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.352851    5028 generic.go:356] "Generic (PLEG): container finished" podID="919be555-ac42-475b-932e-fedb50ebea01" containerID="3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924" exitCode=0
Oct 30 22:49:53.353004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.352900    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"
Oct 30 22:49:53.353004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.352912    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" event={"ID":"919be555-ac42-475b-932e-fedb50ebea01","Type":"ContainerDied","Data":"3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924"}
Oct 30 22:49:53.353004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.352945    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl" event={"ID":"919be555-ac42-475b-932e-fedb50ebea01","Type":"ContainerDied","Data":"d445804953432f8cd19502e61e1018edf561ae44de7b0e2890e7b5b71f811204"}
Oct 30 22:49:53.361654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.361632    5028 scope.go:122] "RemoveContainer" containerID="fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54"
Oct 30 22:49:53.361953 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:53.361934    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54\": container with ID starting with fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54 not found: ID does not exist" containerID="fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54"
Oct 30 22:49:53.362013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.361962    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54"} err="failed to get container status \"fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54\": rpc error: code = NotFound desc = could not find container \"fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54\": container with ID starting with fc1a098edb30a5956652e4adaaddb5d55a4deff5b95e808154f267bc91e2be54 not found: ID does not exist"
Oct 30 22:49:53.362013 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.361982    5028 scope.go:122] "RemoveContainer" containerID="3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924"
Oct 30 22:49:53.370944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.370929    5028 scope.go:122] "RemoveContainer" containerID="3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924"
Oct 30 22:49:53.371220 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:53.371191    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924\": container with ID starting with 3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924 not found: ID does not exist" containerID="3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924"
Oct 30 22:49:53.371286 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.371225    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924"} err="failed to get container status \"3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924\": rpc error: code = NotFound desc = could not find container \"3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924\": container with ID starting with 3d439de8868d9690a43c3ef62894c0e71c6147987cabe298a4c2f75576cb0924 not found: ID does not exist"
Oct 30 22:49:53.526212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.526172    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"]
Oct 30 22:49:53.530985 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.530952    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eb4aa1-76c5f496d7-jh5q2"]
Oct 30 22:49:53.635932 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.635849    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"]
Oct 30 22:49:53.640780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:53.640753    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eb4aa1-685ddbddc-splsl"]
Oct 30 22:49:55.298922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:55.298887    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" path="/var/lib/kubelet/pods/5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a/volumes"
Oct 30 22:49:55.299331 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:55.299267    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919be555-ac42-475b-932e-fedb50ebea01" path="/var/lib/kubelet/pods/919be555-ac42-475b-932e-fedb50ebea01/volumes"
Oct 30 22:49:57.913242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:57.913193    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:58.110948 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:58.110905    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:58.435364 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:49:58.435329    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:49:58.894262 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:49:58.894223    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.487600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:00.487550    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.649604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:00.649575    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.819188 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:00.819106    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"]
Oct 30 22:50:00.819391 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:00.819365    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" containerID="cri-o://4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320" gracePeriod=30
Oct 30 22:50:00.829557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:00.829524    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:01.330445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:01.330407    5028 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:50:01.330628 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:01.330470    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:50:05.117244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:05.117203    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:06.081228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.081201    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:50:06.185922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.185830    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.185922 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.185880    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.185929    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186013    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186042    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186104    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186137    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186165    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186186    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186275    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186333    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token\") pod \"cd9a5089-8749-4d9b-b53d-96eb145c757a\" (UID: \"cd9a5089-8749-4d9b-b53d-96eb145c757a\") "
Oct 30 22:50:06.186500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186364    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:06.187134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186537    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:06.187134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186651    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:06.187134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186748    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.187134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186770    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.187134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186787    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.187134 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.186801    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.188472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.188439    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:06.188592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.188476    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:06.188592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.188482    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:06.188680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.188588    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g" pod "cd9a5089-8749-4d9b-b53d-96eb145c757a" (UID: "cd9a5089-8749-4d9b-b53d-96eb145c757a"). InnerVolumeSpecName "kube-api-access-vkw8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:06.288255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.288216    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkw8g\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-kube-api-access-vkw8g\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.288255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.288250    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.288255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.288260    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.288507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.288269    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9a5089-8749-4d9b-b53d-96eb145c757a-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.288507 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.288281    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9a5089-8749-4d9b-b53d-96eb145c757a-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.442023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.441934    5028 generic.go:356] "Generic (PLEG): container finished" podID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerID="4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320" exitCode=0
Oct 30 22:50:06.442023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.442005    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"
Oct 30 22:50:06.442279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.442020    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" event={"ID":"cd9a5089-8749-4d9b-b53d-96eb145c757a","Type":"ContainerDied","Data":"4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320"}
Oct 30 22:50:06.442279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.442085    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm" event={"ID":"cd9a5089-8749-4d9b-b53d-96eb145c757a","Type":"ContainerDied","Data":"a69322ce1f4c89153c7959ce2201c1cffa2e1cf2edf557262c62057cf6d6ebea"}
Oct 30 22:50:06.442279 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.442114    5028 scope.go:122] "RemoveContainer" containerID="4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320"
Oct 30 22:50:06.452839 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.452764    5028 scope.go:122] "RemoveContainer" containerID="9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"
Oct 30 22:50:06.462247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.462223    5028 scope.go:122] "RemoveContainer" containerID="4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320"
Oct 30 22:50:06.462533 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:50:06.462515    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320\": container with ID starting with 4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320 not found: ID does not exist" containerID="4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320"
Oct 30 22:50:06.462620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.462540    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320"} err="failed to get container status \"4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320\": rpc error: code = NotFound desc = could not find container \"4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320\": container with ID starting with 4e6a10a5f479072412dbde01d6e0f696d5281bf920001acf542952c01f8c4320 not found: ID does not exist"
Oct 30 22:50:06.462620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.462557    5028 scope.go:122] "RemoveContainer" containerID="9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"
Oct 30 22:50:06.462839 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:50:06.462823    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9\": container with ID starting with 9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9 not found: ID does not exist" containerID="9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"
Oct 30 22:50:06.462885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.462844    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9"} err="failed to get container status \"9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9\": rpc error: code = NotFound desc = could not find container \"9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9\": container with ID starting with 9f22bea2114b7102e8203956b0658d03de66fe25af47c7500c35e268fa8406a9 not found: ID does not exist"
Oct 30 22:50:06.606257 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.606208    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"]
Oct 30 22:50:06.610802 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:06.610772    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-qvjpm"]
Oct 30 22:50:07.289675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:07.289643    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" path="/var/lib/kubelet/pods/cd9a5089-8749-4d9b-b53d-96eb145c757a/volumes"
Oct 30 22:50:16.680946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:16.680910    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:19.208736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:19.208701    5028 scope.go:122] "RemoveContainer" containerID="80b8d23b67038ec38be0a96b54a8bab6a2fe1104a6c2a327309874c9d6dc7ced"
Oct 30 22:50:27.061492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:27.061453    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"]
Oct 30 22:50:27.108361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:27.108321    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cec3a832-5kp5c"]
Oct 30 22:50:27.289822 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:27.289791    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052aae37-53a3-4ea1-86a7-ca267534e209" path="/var/lib/kubelet/pods/052aae37-53a3-4ea1-86a7-ca267534e209/volumes"
Oct 30 22:50:28.436025 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:50:28.435988    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:50:29.944869 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.944828    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"]
Oct 30 22:50:29.945268 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945256    5028 cpu_manager.go:414] "containerMap: removing container" podUID="919be555-ac42-475b-932e-fedb50ebea01" containerName="payload-pre-processing"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945267    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919be555-ac42-475b-932e-fedb50ebea01" containerName="payload-pre-processing"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945280    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerName="payload-processing"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945285    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerName="payload-processing"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945294    5028 cpu_manager.go:414] "containerMap: removing container" podUID="67dd0114-0c8c-4bab-83fe-e6df2c18175c" containerName="revoke-keys"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945299    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dd0114-0c8c-4bab-83fe-e6df2c18175c" containerName="revoke-keys"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945307    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy"
Oct 30 22:50:29.945311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945312    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945321    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945325    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945336    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945340    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b27184-4cbd-46d1-ade8-c061d5da46a4" containerName="istio-proxy"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945435    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="919be555-ac42-475b-932e-fedb50ebea01" containerName="payload-pre-processing"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945443    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5283cbc2-9bf7-4db4-9bf5-19fee8c1df8a" containerName="payload-processing"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945449    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dd0114-0c8c-4bab-83fe-e6df2c18175c" containerName="revoke-keys"
Oct 30 22:50:29.945586 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.945455    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy"
Oct 30 22:50:29.950247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:29.950221    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.001661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.001631    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:30.006363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.006332    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7swf\" (UniqueName: \"kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.006508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.006393    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.006508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.006459    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.048494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.048457    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:30.048494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.048457    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:30.048494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.048464    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:50:30.051473 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.051451    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"]
Oct 30 22:50:30.107531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.107495    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.107769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.107620    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7swf\" (UniqueName: \"kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.107769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.107673    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.108348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.108325    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.110038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.110013    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.166675 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.166644    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7swf\" (UniqueName: \"kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf\") pod \"maas-api-revoke-sub-85237e45-pb758\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.263350 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.263315    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:30.636845 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:30.636819    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"]
Oct 30 22:50:30.638338 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:50:30.638312    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a259025_215f_4c31_a544_ed87b7b08de1.slice/crio-04413dcc105cf8f8c39bfd6e1bd19baf636b07ddd6c05788a326b0a6d5eb5b9b WatchSource:0}: Error finding container 04413dcc105cf8f8c39bfd6e1bd19baf636b07ddd6c05788a326b0a6d5eb5b9b: Status 404 returned error can't find the container with id 04413dcc105cf8f8c39bfd6e1bd19baf636b07ddd6c05788a326b0a6d5eb5b9b
Oct 30 22:50:31.601244 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:31.601206    5028 generic.go:356] "Generic (PLEG): container finished" podID="4a259025-215f-4c31-a544-ed87b7b08de1" containerID="92b8ee8007126c4e6773ec2cdcf19aff27b9e9f296cd40aa53dad4938da34367" exitCode=0
Oct 30 22:50:31.601637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:31.601294    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758" event={"ID":"4a259025-215f-4c31-a544-ed87b7b08de1","Type":"ContainerDied","Data":"92b8ee8007126c4e6773ec2cdcf19aff27b9e9f296cd40aa53dad4938da34367"}
Oct 30 22:50:31.601637 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:31.601346    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758" event={"ID":"4a259025-215f-4c31-a544-ed87b7b08de1","Type":"ContainerStarted","Data":"04413dcc105cf8f8c39bfd6e1bd19baf636b07ddd6c05788a326b0a6d5eb5b9b"}
Oct 30 22:50:32.743972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.743945    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:32.835363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.835324    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf\" (UniqueName: \"kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf\") pod \"4a259025-215f-4c31-a544-ed87b7b08de1\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") "
Oct 30 22:50:32.835554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.835391    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca\") pod \"4a259025-215f-4c31-a544-ed87b7b08de1\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") "
Oct 30 22:50:32.835554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.835425    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca\") pod \"4a259025-215f-4c31-a544-ed87b7b08de1\" (UID: \"4a259025-215f-4c31-a544-ed87b7b08de1\") "
Oct 30 22:50:32.835792 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.835766    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca" pod "4a259025-215f-4c31-a544-ed87b7b08de1" (UID: "4a259025-215f-4c31-a544-ed87b7b08de1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:32.837593 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.837571    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca" pod "4a259025-215f-4c31-a544-ed87b7b08de1" (UID: "4a259025-215f-4c31-a544-ed87b7b08de1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:32.837658 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.837587    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf" pod "4a259025-215f-4c31-a544-ed87b7b08de1" (UID: "4a259025-215f-4c31-a544-ed87b7b08de1"). InnerVolumeSpecName "kube-api-access-x7swf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:32.937003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.936912    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:32.937003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.936945    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a259025-215f-4c31-a544-ed87b7b08de1-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:32.937003 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:32.936955    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7swf\" (UniqueName: \"kubernetes.io/projected/4a259025-215f-4c31-a544-ed87b7b08de1-kube-api-access-x7swf\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:33.615425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:33.615345    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"
Oct 30 22:50:33.615425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:33.615386    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758" event={"ID":"4a259025-215f-4c31-a544-ed87b7b08de1","Type":"ContainerDied","Data":"04413dcc105cf8f8c39bfd6e1bd19baf636b07ddd6c05788a326b0a6d5eb5b9b"}
Oct 30 22:50:33.615621 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:33.615437    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04413dcc105cf8f8c39bfd6e1bd19baf636b07ddd6c05788a326b0a6d5eb5b9b"
Oct 30 22:50:40.049280 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.049243    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"]
Oct 30 22:50:40.096411 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.096372    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e240c47a-dljhs"]
Oct 30 22:50:40.258653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.258616    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"]
Oct 30 22:50:40.259106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.259092    5028 cpu_manager.go:414] "containerMap: removing container" podUID="4a259025-215f-4c31-a544-ed87b7b08de1" containerName="revoke-keys"
Oct 30 22:50:40.259154 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.259107    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a259025-215f-4c31-a544-ed87b7b08de1" containerName="revoke-keys"
Oct 30 22:50:40.259191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.259167    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9a5089-8749-4d9b-b53d-96eb145c757a" containerName="istio-proxy"
Oct 30 22:50:40.259191 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.259176    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a259025-215f-4c31-a544-ed87b7b08de1" containerName="revoke-keys"
Oct 30 22:50:40.262361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.262344    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302033    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mff6\" (UniqueName: \"kubernetes.io/projected/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-kube-api-access-6mff6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302101    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302500 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302460    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302530    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302638 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302600    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302710    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302744    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302799 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302764    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.302995 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.302844    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.341972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.341938    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"]
Oct 30 22:50:40.403971 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.403924    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mff6\" (UniqueName: \"kubernetes.io/projected/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-kube-api-access-6mff6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.403982    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404018    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404041    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404176 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404087    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404228    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404281    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404312    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404419 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404338    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404545    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404610    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404709    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404740    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.404934 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.404917    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.406585 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.406561    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.406787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.406767    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.462676 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.462649    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.462907 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.462884    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mff6\" (UniqueName: \"kubernetes.io/projected/f1f27ceb-9be0-407c-a559-b5e3338ad1d3-kube-api-access-6mff6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-5c66s\" (UID: \"f1f27ceb-9be0-407c-a559-b5e3338ad1d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.575309 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.575217    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:40.992121 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.992094    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"]
Oct 30 22:50:40.993179 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:50:40.993155    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f27ceb_9be0_407c_a559_b5e3338ad1d3.slice/crio-068868c6d3b2a48a8ea39e3185d4b276d8d3e01004314fb71a148a3ffbcfc141 WatchSource:0}: Error finding container 068868c6d3b2a48a8ea39e3185d4b276d8d3e01004314fb71a148a3ffbcfc141: Status 404 returned error can't find the container with id 068868c6d3b2a48a8ea39e3185d4b276d8d3e01004314fb71a148a3ffbcfc141
Oct 30 22:50:40.995285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.995253    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:50:40.995337 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.995319    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:50:40.995380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:40.995348    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:50:41.291033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:41.290953    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f0990d-b0b9-4c14-8916-0d6ccb109431" path="/var/lib/kubelet/pods/f1f0990d-b0b9-4c14-8916-0d6ccb109431/volumes"
Oct 30 22:50:41.667249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:41.667161    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" event={"ID":"f1f27ceb-9be0-407c-a559-b5e3338ad1d3","Type":"ContainerStarted","Data":"960c248c000b5ac4fb0fe80862c4496c6b1ce238e61fca12e18e744e673b2e13"}
Oct 30 22:50:41.667249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:41.667221    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" event={"ID":"f1f27ceb-9be0-407c-a559-b5e3338ad1d3","Type":"ContainerStarted","Data":"068868c6d3b2a48a8ea39e3185d4b276d8d3e01004314fb71a148a3ffbcfc141"}
Oct 30 22:50:41.791285 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:41.791219    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podStartSLOduration=1.791205103 podStartE2EDuration="1.791205103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:41.789404581 +0000 UTC m=+2972.638917555" watchObservedRunningTime="2026-10-30 22:50:41.791205103 +0000 UTC m=+2972.640718075"
Oct 30 22:50:42.483668 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:42.483631    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:42.575661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:42.575625    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:42.577390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:42.577365    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:42.577515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:42.577424    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:43.576425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:43.576383    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:43.576874 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:43.576440    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:44.576040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:44.575998    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:44.576212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:44.576060    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:44.770697 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:44.770651    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:45.576445 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:45.576412    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:45.576651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:45.576470    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:46.576204 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:46.576172    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:46.576627 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:46.576227    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:47.576329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:47.576285    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:47.576745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:47.576357    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:48.575784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:48.575750    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-5c66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 30 22:50:48.576079 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:48.575803    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s" podUID="f1f27ceb-9be0-407c-a559-b5e3338ad1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 30 22:50:49.579450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:49.579415    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:49.579962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:49.579726    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:49.580508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:50:49.580491    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-5c66s"
Oct 30 22:50:58.435183 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:50:58.435143    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:51:08.051136 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:08.051088    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"]
Oct 30 22:51:08.098588 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:08.098539    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"]
Oct 30 22:51:08.101060 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:08.101038    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6543fbd-mfqh4"]
Oct 30 22:51:08.103190 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:08.103164    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-358ebcb9-qk2fc"]
Oct 30 22:51:09.290779 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:09.290746    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b606d7-09d8-4a9d-97a5-6697722ead44" path="/var/lib/kubelet/pods/78b606d7-09d8-4a9d-97a5-6697722ead44/volumes"
Oct 30 22:51:09.291173 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:09.291158    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56b1a0d-e240-49b9-9eb2-2127e876a5c8" path="/var/lib/kubelet/pods/d56b1a0d-e240-49b9-9eb2-2127e876a5c8/volumes"
Oct 30 22:51:10.515453 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:10.515425    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:51:10.560338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:10.560313    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:51:17.127935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:17.127865    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:21.069765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:21.069663    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"]
Oct 30 22:51:21.116673 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:21.116633    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3fd866c-zv2p7"]
Oct 30 22:51:21.289634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:21.289601    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f14f08-635d-4bd0-9c5b-32a5f62fc207" path="/var/lib/kubelet/pods/b1f14f08-635d-4bd0-9c5b-32a5f62fc207/volumes"
Oct 30 22:51:28.435875 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:51:28.435836    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:51:38.055888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:38.055847    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"]
Oct 30 22:51:38.101825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:38.101792    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e5e902-6lldw"]
Oct 30 22:51:39.077235 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.077205    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"]
Oct 30 22:51:39.080883 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.080864    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.133612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.133578    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:39.180591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.180561    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:39.180591 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.180580    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:51:39.180833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.180581    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:39.183463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.183432    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"]
Oct 30 22:51:39.199355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.199322    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.199484 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.199454    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mck\" (UniqueName: \"kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.199532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.199481    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.289424 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.289383    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0790b25-cf84-4869-86a8-398ae57794b3" path="/var/lib/kubelet/pods/a0790b25-cf84-4869-86a8-398ae57794b3/volumes"
Oct 30 22:51:39.300062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.300035    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mck\" (UniqueName: \"kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.300219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.300066    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.300219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.300125    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.300880 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.300862    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.302573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.302546    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.359385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.359311    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mck\" (UniqueName: \"kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck\") pod \"maas-api-revoke-sub-c1e271a9-ttlzn\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.393797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.393764    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:39.778403 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:39.778369    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"]
Oct 30 22:51:39.779751 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:51:39.779724    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b9032_3182_4762_bd8b_0ac547ba5d68.slice/crio-e9b73770e63b171240332ee408fe59c609ca8fc5665242cd78e5097da4ffc0db WatchSource:0}: Error finding container e9b73770e63b171240332ee408fe59c609ca8fc5665242cd78e5097da4ffc0db: Status 404 returned error can't find the container with id e9b73770e63b171240332ee408fe59c609ca8fc5665242cd78e5097da4ffc0db
Oct 30 22:51:40.045819 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.045786    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn" event={"ID":"417b9032-3182-4762-bd8b-0ac547ba5d68","Type":"ContainerStarted","Data":"e9b73770e63b171240332ee408fe59c609ca8fc5665242cd78e5097da4ffc0db"}
Oct 30 22:51:40.705358 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.705321    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"]
Oct 30 22:51:40.708993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.708973    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.752797 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.752768    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"]
Oct 30 22:51:40.756448 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.756426    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.800892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.800858    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"]
Oct 30 22:51:40.817757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.817725    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.817944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.817764    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b61aded-a35f-42a1-8753-462f487ccfef-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.817944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.817789    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.817944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.817889    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.818080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.817986    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.818080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.818018    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b61aded-a35f-42a1-8753-462f487ccfef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.818080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.818052    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b61aded-a35f-42a1-8753-462f487ccfef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.818080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.818074    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.818239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.818192    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q2lm\" (UniqueName: \"kubernetes.io/projected/7b61aded-a35f-42a1-8753-462f487ccfef-kube-api-access-4q2lm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.878465 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.878381    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"]
Oct 30 22:51:40.919092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919053    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b61aded-a35f-42a1-8753-462f487ccfef-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919098    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919128    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919162    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919191    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b61aded-a35f-42a1-8753-462f487ccfef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919225    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a143fab9-aead-4204-80e5-c8a690c95513-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919250    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919282    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b61aded-a35f-42a1-8753-462f487ccfef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919345    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919398    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919427    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a143fab9-aead-4204-80e5-c8a690c95513-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919466    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919509    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919542    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919593    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwssg\" (UniqueName: \"kubernetes.io/projected/a143fab9-aead-4204-80e5-c8a690c95513-kube-api-access-bwssg\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.919644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919620    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a143fab9-aead-4204-80e5-c8a690c95513-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:40.920103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919652    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q2lm\" (UniqueName: \"kubernetes.io/projected/7b61aded-a35f-42a1-8753-462f487ccfef-kube-api-access-4q2lm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.920103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919662    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.920103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919648    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.920103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919667    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.920103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.919842    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.920304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.920124    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b61aded-a35f-42a1-8753-462f487ccfef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.920304 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.920129    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.921555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.921533    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b61aded-a35f-42a1-8753-462f487ccfef-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.921847 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.921829    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b61aded-a35f-42a1-8753-462f487ccfef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.978236 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.978202    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b61aded-a35f-42a1-8753-462f487ccfef-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:40.978400 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:40.978358    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q2lm\" (UniqueName: \"kubernetes.io/projected/7b61aded-a35f-42a1-8753-462f487ccfef-kube-api-access-4q2lm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-dgkjv\" (UID: \"7b61aded-a35f-42a1-8753-462f487ccfef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:41.021230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021191    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a143fab9-aead-4204-80e5-c8a690c95513-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021230 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021228    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021254    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021276    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a143fab9-aead-4204-80e5-c8a690c95513-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021303    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021325    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021349    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021460    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwssg\" (UniqueName: \"kubernetes.io/projected/a143fab9-aead-4204-80e5-c8a690c95513-kube-api-access-bwssg\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021488    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a143fab9-aead-4204-80e5-c8a690c95513-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021601    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021828 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021671    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.021942 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.021851    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:41.022217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.022190    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a143fab9-aead-4204-80e5-c8a690c95513-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.022324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.022229    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.022324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.022258    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.024008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.023990    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a143fab9-aead-4204-80e5-c8a690c95513-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.024062 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.024023    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a143fab9-aead-4204-80e5-c8a690c95513-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.054406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.054376    5028 generic.go:356] "Generic (PLEG): container finished" podID="417b9032-3182-4762-bd8b-0ac547ba5d68" containerID="1950c900803297cac6dd049a7e344c1923f63bf166298bbf85574b4af48287a4" exitCode=0
Oct 30 22:51:41.054573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.054437    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn" event={"ID":"417b9032-3182-4762-bd8b-0ac547ba5d68","Type":"ContainerDied","Data":"1950c900803297cac6dd049a7e344c1923f63bf166298bbf85574b4af48287a4"}
Oct 30 22:51:41.081739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.081372    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a143fab9-aead-4204-80e5-c8a690c95513-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.081739 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.081591    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwssg\" (UniqueName: \"kubernetes.io/projected/a143fab9-aead-4204-80e5-c8a690c95513-kube-api-access-bwssg\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rwbtf\" (UID: \"a143fab9-aead-4204-80e5-c8a690c95513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.369248 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.369212    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:41.460964 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:51:41.460916    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b61aded_a35f_42a1_8753_462f487ccfef.slice/crio-408245f0867e754fe4f4d22a6ec55ddd27fd32a379e002a350f4e28498929edf WatchSource:0}: Error finding container 408245f0867e754fe4f4d22a6ec55ddd27fd32a379e002a350f4e28498929edf: Status 404 returned error can't find the container with id 408245f0867e754fe4f4d22a6ec55ddd27fd32a379e002a350f4e28498929edf
Oct 30 22:51:41.462952 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.462924    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"]
Oct 30 22:51:41.463950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.463913    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:41.464026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.464000    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:41.464107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.464052    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:41.745726 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.745672    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"]
Oct 30 22:51:41.746178 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:51:41.746037    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda143fab9_aead_4204_80e5_c8a690c95513.slice/crio-8b1640ae1eaf7f9783d19fd462f95027ebb71a198444b73728e600ef7760c276 WatchSource:0}: Error finding container 8b1640ae1eaf7f9783d19fd462f95027ebb71a198444b73728e600ef7760c276: Status 404 returned error can't find the container with id 8b1640ae1eaf7f9783d19fd462f95027ebb71a198444b73728e600ef7760c276
Oct 30 22:51:41.748514 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.748479    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:41.748634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.748539    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:41.748634 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:41.748570    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:51:42.063030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.062932    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" event={"ID":"a143fab9-aead-4204-80e5-c8a690c95513","Type":"ContainerStarted","Data":"24fb47cc67f7949f2ae9cf2d94091008a49f46e51d0ac96b0a457902fc576bf0"}
Oct 30 22:51:42.063030 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.062997    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" event={"ID":"a143fab9-aead-4204-80e5-c8a690c95513","Type":"ContainerStarted","Data":"8b1640ae1eaf7f9783d19fd462f95027ebb71a198444b73728e600ef7760c276"}
Oct 30 22:51:42.065261 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.065231    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" event={"ID":"7b61aded-a35f-42a1-8753-462f487ccfef","Type":"ContainerStarted","Data":"40e8c9b19c0c4e6392aab9626f2c4d559ef6e5826d8057c635c9aed390d90060"}
Oct 30 22:51:42.065426 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.065278    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" event={"ID":"7b61aded-a35f-42a1-8753-462f487ccfef","Type":"ContainerStarted","Data":"408245f0867e754fe4f4d22a6ec55ddd27fd32a379e002a350f4e28498929edf"}
Oct 30 22:51:42.215652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.215600    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podStartSLOduration=2.215582323 podStartE2EDuration="2.215582323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.212732378 +0000 UTC m=+3033.062245352" watchObservedRunningTime="2026-10-30 22:51:42.215582323 +0000 UTC m=+3033.065095297"
Oct 30 22:51:42.254022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.253981    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:42.345807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.345165    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca\") pod \"417b9032-3182-4762-bd8b-0ac547ba5d68\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") "
Oct 30 22:51:42.345807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.345227    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck\" (UniqueName: \"kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck\") pod \"417b9032-3182-4762-bd8b-0ac547ba5d68\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") "
Oct 30 22:51:42.345807 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.345306    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca\") pod \"417b9032-3182-4762-bd8b-0ac547ba5d68\" (UID: \"417b9032-3182-4762-bd8b-0ac547ba5d68\") "
Oct 30 22:51:42.347773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.347710    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podStartSLOduration=2.347677077 podStartE2EDuration="2.347677077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.346638632 +0000 UTC m=+3033.196151605" watchObservedRunningTime="2026-10-30 22:51:42.347677077 +0000 UTC m=+3033.197190051"
Oct 30 22:51:42.348363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.348329    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca" pod "417b9032-3182-4762-bd8b-0ac547ba5d68" (UID: "417b9032-3182-4762-bd8b-0ac547ba5d68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:42.350052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.349910    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca" pod "417b9032-3182-4762-bd8b-0ac547ba5d68" (UID: "417b9032-3182-4762-bd8b-0ac547ba5d68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:42.352546 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.352518    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck" pod "417b9032-3182-4762-bd8b-0ac547ba5d68" (UID: "417b9032-3182-4762-bd8b-0ac547ba5d68"). InnerVolumeSpecName "kube-api-access-m9mck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:42.371730 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.371188    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:42.373361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.372932    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:42.373361 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.373000    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:42.449711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.449259    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.449711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.449291    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mck\" (UniqueName: \"kubernetes.io/projected/417b9032-3182-4762-bd8b-0ac547ba5d68-kube-api-access-m9mck\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:42.449711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:42.449322    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b9032-3182-4762-bd8b-0ac547ba5d68-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.037712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.024064    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:43.037712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.026205    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:43.037712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.026262    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:43.077214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.077183    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"
Oct 30 22:51:43.077410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.077221    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn" event={"ID":"417b9032-3182-4762-bd8b-0ac547ba5d68","Type":"ContainerDied","Data":"e9b73770e63b171240332ee408fe59c609ca8fc5665242cd78e5097da4ffc0db"}
Oct 30 22:51:43.077410 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.077290    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b73770e63b171240332ee408fe59c609ca8fc5665242cd78e5097da4ffc0db"
Oct 30 22:51:43.370487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.370393    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:43.370760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:43.370726    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:44.029713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:44.026164    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:44.029713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:44.026232    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:44.370394 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:44.370297    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:44.371032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:44.370999    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:45.036239 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:45.036202    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:45.036547 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:45.036514    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:45.369986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:45.369882    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:45.370183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:45.370071    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:46.025217 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:46.025173    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:46.025862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:46.025827    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:46.369720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:46.369610    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:46.369945 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:46.369924    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:47.025149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:47.025107    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:47.025464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:47.025433    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:47.370860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:47.370752    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:47.370860 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:47.370834    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:48.022618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:48.022567    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:48.022872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:48.022635    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:48.370611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:48.370507    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:48.370611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:48.370564    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:49.022889 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:49.022852    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-dgkjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 30 22:51:49.023070 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:49.022905    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv" podUID="7b61aded-a35f-42a1-8753-462f487ccfef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 30 22:51:49.370608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:49.370518    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 30 22:51:49.370608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:49.370586    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 30 22:51:50.027294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:50.027260    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:50.028071 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:50.028045    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:50.028642 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:50.028619    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-dgkjv"
Oct 30 22:51:50.373930 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:50.373840    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:50.374468 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:50.374126    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:50.374943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:51:50.374916    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf"
Oct 30 22:51:58.475471 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:51:58.475260    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:52:10.421284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.421245    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"]
Oct 30 22:52:10.421709 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.421697    5028 cpu_manager.go:414] "containerMap: removing container" podUID="417b9032-3182-4762-bd8b-0ac547ba5d68" containerName="revoke-keys"
Oct 30 22:52:10.421765 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.421710    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417b9032-3182-4762-bd8b-0ac547ba5d68" containerName="revoke-keys"
Oct 30 22:52:10.421806 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.421796    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="417b9032-3182-4762-bd8b-0ac547ba5d68" containerName="revoke-keys"
Oct 30 22:52:10.425015 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.424994    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.482363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.482329    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"]
Oct 30 22:52:10.502100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.502064    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.502100 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.502100    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.502310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.502251    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5nl\" (UniqueName: \"kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.525113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.525071    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:10.525113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.525106    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:52:10.525324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.525139    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:10.525324 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.525138    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:10.603161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.603128    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.603345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.603248    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5nl\" (UniqueName: \"kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.603345 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.603293    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.604023 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.603995    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.605558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.605529    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.667713 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.667669    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5nl\" (UniqueName: \"kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl\") pod \"maas-api-revoke-sub-2e1f5b5f-9xkns\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:10.738578 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:10.738539    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:11.110074 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:11.110046    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"]
Oct 30 22:52:11.111217 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:52:11.111184    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19088ec_f220_4a4f_b5dc_347d277e2127.slice/crio-85454f656e06d6ef47ca0ad7ad8ca69e29d6647ced3cc137c583a098c85ac267 WatchSource:0}: Error finding container 85454f656e06d6ef47ca0ad7ad8ca69e29d6647ced3cc137c583a098c85ac267: Status 404 returned error can't find the container with id 85454f656e06d6ef47ca0ad7ad8ca69e29d6647ced3cc137c583a098c85ac267
Oct 30 22:52:11.341372 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:11.341343    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns" event={"ID":"a19088ec-f220-4a4f-b5dc-347d277e2127","Type":"ContainerStarted","Data":"85454f656e06d6ef47ca0ad7ad8ca69e29d6647ced3cc137c583a098c85ac267"}
Oct 30 22:52:12.350438 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:12.350347    5028 generic.go:356] "Generic (PLEG): container finished" podID="a19088ec-f220-4a4f-b5dc-347d277e2127" containerID="22b5cdb1cfdae8f29f8631d4f01103c46268716320be0b1baaae00ce0f8a98fa" exitCode=0
Oct 30 22:52:12.350872 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:12.350431    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns" event={"ID":"a19088ec-f220-4a4f-b5dc-347d277e2127","Type":"ContainerDied","Data":"22b5cdb1cfdae8f29f8631d4f01103c46268716320be0b1baaae00ce0f8a98fa"}
Oct 30 22:52:13.504943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.504914    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:13.533144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.533114    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca\") pod \"a19088ec-f220-4a4f-b5dc-347d277e2127\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") "
Oct 30 22:52:13.533294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.533152    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca\") pod \"a19088ec-f220-4a4f-b5dc-347d277e2127\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") "
Oct 30 22:52:13.533294 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.533221    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl\" (UniqueName: \"kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl\") pod \"a19088ec-f220-4a4f-b5dc-347d277e2127\" (UID: \"a19088ec-f220-4a4f-b5dc-347d277e2127\") "
Oct 30 22:52:13.533604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.533568    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca" pod "a19088ec-f220-4a4f-b5dc-347d277e2127" (UID: "a19088ec-f220-4a4f-b5dc-347d277e2127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:13.535371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.535343    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca" pod "a19088ec-f220-4a4f-b5dc-347d277e2127" (UID: "a19088ec-f220-4a4f-b5dc-347d277e2127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:13.535534 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.535511    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl" pod "a19088ec-f220-4a4f-b5dc-347d277e2127" (UID: "a19088ec-f220-4a4f-b5dc-347d277e2127"). InnerVolumeSpecName "kube-api-access-2x5nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:13.634265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.634183    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x5nl\" (UniqueName: \"kubernetes.io/projected/a19088ec-f220-4a4f-b5dc-347d277e2127-kube-api-access-2x5nl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:13.634265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.634222    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:13.634265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:13.634238    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19088ec-f220-4a4f-b5dc-347d277e2127-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:14.370652 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:14.370621    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"
Oct 30 22:52:14.370843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:14.370620    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns" event={"ID":"a19088ec-f220-4a4f-b5dc-347d277e2127","Type":"ContainerDied","Data":"85454f656e06d6ef47ca0ad7ad8ca69e29d6647ced3cc137c583a098c85ac267"}
Oct 30 22:52:14.370843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:14.370748    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85454f656e06d6ef47ca0ad7ad8ca69e29d6647ced3cc137c583a098c85ac267"
Oct 30 22:52:17.389019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:17.388984    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.274551 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.274516    5028 scope.go:122] "RemoveContainer" containerID="8989a5ee30cc9be641978f5da8406e2c0fb6e6607e7b505d49091784b378c871"
Oct 30 22:52:19.284527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.284493    5028 scope.go:122] "RemoveContainer" containerID="274bbbfe6f1fe60b285117dc8f2b071b480050b35d9a36a560ab8e78e30173a1"
Oct 30 22:52:19.293789 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.293731    5028 scope.go:122] "RemoveContainer" containerID="8bb0f9298ff0c23596a195a555678033ad93dbf8c8c0f4147dc9f5758ef03b93"
Oct 30 22:52:19.302631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.302613    5028 scope.go:122] "RemoveContainer" containerID="1f0c1866acec303155a81d25184a84dfd7375f6e7bad36c043962d897f129b22"
Oct 30 22:52:19.311371 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.311346    5028 scope.go:122] "RemoveContainer" containerID="5798268c1eb85280656e6564f6967543fc3d934168776214f93458a55bbedad5"
Oct 30 22:52:19.320050 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.320022    5028 scope.go:122] "RemoveContainer" containerID="c0c3aeecb48304c327789641ced45271fbd7abff149e35051ff7e775ef424568"
Oct 30 22:52:19.328464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:19.328445    5028 scope.go:122] "RemoveContainer" containerID="10e229ba72c9ebae851aa9c06e7810da0fb059a99bdb6ddab3e5386a3e6acc63"
Oct 30 22:52:22.826768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.826734    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"]
Oct 30 22:52:22.827423 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.827404    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a19088ec-f220-4a4f-b5dc-347d277e2127" containerName="revoke-keys"
Oct 30 22:52:22.827477 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.827427    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19088ec-f220-4a4f-b5dc-347d277e2127" containerName="revoke-keys"
Oct 30 22:52:22.827557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.827546    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19088ec-f220-4a4f-b5dc-347d277e2127" containerName="revoke-keys"
Oct 30 22:52:22.832698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.832672    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:22.886925 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.886893    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:22.934125 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.934093    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:22.934317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.934139    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:22.934317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.934266    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246d4\" (UniqueName: \"kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:22.934520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.934503    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:52:22.934520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.934514    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:22.937296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.937274    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"]
Oct 30 22:52:22.976150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:22.976115    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:23.035523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.035485    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246d4\" (UniqueName: \"kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.035749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.035598    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.035749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.035643    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.036315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.036291    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.038189 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.038166    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.095284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.095200    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-246d4\" (UniqueName: \"kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4\") pod \"maas-api-revoke-sub-ec843018-c9lbl\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.146073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.146035    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:23.522272 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:23.522238    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"]
Oct 30 22:52:23.523592 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:52:23.523566    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf624e487_f8b8_4e3a_a885_0235614d3e68.slice/crio-6faa19d229a51364de1292f0e2cc5db59d06f9357974e5aa30265ab9fccf40ee WatchSource:0}: Error finding container 6faa19d229a51364de1292f0e2cc5db59d06f9357974e5aa30265ab9fccf40ee: Status 404 returned error can't find the container with id 6faa19d229a51364de1292f0e2cc5db59d06f9357974e5aa30265ab9fccf40ee
Oct 30 22:52:24.143504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:24.143318    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:24.304744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:24.304709    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:24.434760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:24.434646    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl" event={"ID":"f624e487-f8b8-4e3a-a885-0235614d3e68","Type":"ContainerStarted","Data":"348381be5759955917e4ffad256a19fbae17d1dc9c7d6213c490228bfb6606a8"}
Oct 30 22:52:24.434760 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:24.434722    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl" event={"ID":"f624e487-f8b8-4e3a-a885-0235614d3e68","Type":"ContainerStarted","Data":"6faa19d229a51364de1292f0e2cc5db59d06f9357974e5aa30265ab9fccf40ee"}
Oct 30 22:52:24.547671 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:24.547616    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl" podStartSLOduration=2.547601382 podStartE2EDuration="2.547601382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:24.546145807 +0000 UTC m=+3075.395658781" watchObservedRunningTime="2026-10-30 22:52:24.547601382 +0000 UTC m=+3075.397114349"
Oct 30 22:52:25.442818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:25.442782    5028 generic.go:356] "Generic (PLEG): container finished" podID="f624e487-f8b8-4e3a-a885-0235614d3e68" containerID="348381be5759955917e4ffad256a19fbae17d1dc9c7d6213c490228bfb6606a8" exitCode=0
Oct 30 22:52:25.443249 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:25.442875    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl" event={"ID":"f624e487-f8b8-4e3a-a885-0235614d3e68","Type":"ContainerDied","Data":"348381be5759955917e4ffad256a19fbae17d1dc9c7d6213c490228bfb6606a8"}
Oct 30 22:52:26.591906 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.591877    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:26.673996 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.673957    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca\") pod \"f624e487-f8b8-4e3a-a885-0235614d3e68\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") "
Oct 30 22:52:26.674199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.674053    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca\") pod \"f624e487-f8b8-4e3a-a885-0235614d3e68\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") "
Oct 30 22:52:26.674199 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.674164    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4\" (UniqueName: \"kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4\") pod \"f624e487-f8b8-4e3a-a885-0235614d3e68\" (UID: \"f624e487-f8b8-4e3a-a885-0235614d3e68\") "
Oct 30 22:52:26.674437 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.674413    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca" pod "f624e487-f8b8-4e3a-a885-0235614d3e68" (UID: "f624e487-f8b8-4e3a-a885-0235614d3e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:26.676172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.676147    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4" pod "f624e487-f8b8-4e3a-a885-0235614d3e68" (UID: "f624e487-f8b8-4e3a-a885-0235614d3e68"). InnerVolumeSpecName "kube-api-access-246d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:26.676228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.676158    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca" pod "f624e487-f8b8-4e3a-a885-0235614d3e68" (UID: "f624e487-f8b8-4e3a-a885-0235614d3e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:26.775258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.775224    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246d4\" (UniqueName: \"kubernetes.io/projected/f624e487-f8b8-4e3a-a885-0235614d3e68-kube-api-access-246d4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:26.775258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.775255    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:26.775258 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:26.775265    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f624e487-f8b8-4e3a-a885-0235614d3e68-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:27.457118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:27.457085    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"
Oct 30 22:52:27.457310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:27.457122    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl" event={"ID":"f624e487-f8b8-4e3a-a885-0235614d3e68","Type":"ContainerDied","Data":"6faa19d229a51364de1292f0e2cc5db59d06f9357974e5aa30265ab9fccf40ee"}
Oct 30 22:52:27.457310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:27.457181    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6faa19d229a51364de1292f0e2cc5db59d06f9357974e5aa30265ab9fccf40ee"
Oct 30 22:52:28.436459 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:52:28.436428    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:52:36.049404 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:36.049358    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"]
Oct 30 22:52:36.096520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:36.096486    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c7ae23-nbk9l"]
Oct 30 22:52:36.098887 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:36.098872    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"]
Oct 30 22:52:36.100962 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:36.100938    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c6ff414-skhl6"]
Oct 30 22:52:37.289680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:37.289646    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52308cd1-95c8-4146-a8d7-d6daa7588c28" path="/var/lib/kubelet/pods/52308cd1-95c8-4146-a8d7-d6daa7588c28/volumes"
Oct 30 22:52:37.290118 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:37.290077    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85beee0-64bb-436c-96ef-0b60708a98f5" path="/var/lib/kubelet/pods/b85beee0-64bb-436c-96ef-0b60708a98f5/volumes"
Oct 30 22:52:38.998548 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:38.998519    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.471711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:43.468568    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:58.395211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.395126    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"]
Oct 30 22:52:58.395620 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.395604    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f624e487-f8b8-4e3a-a885-0235614d3e68" containerName="revoke-keys"
Oct 30 22:52:58.395667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.395622    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f624e487-f8b8-4e3a-a885-0235614d3e68" containerName="revoke-keys"
Oct 30 22:52:58.395758 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.395748    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f624e487-f8b8-4e3a-a885-0235614d3e68" containerName="revoke-keys"
Oct 30 22:52:58.398719 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.398682    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.434817 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:52:58.434781    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:52:58.450862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.450827    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:58.454399 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.454358    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"]
Oct 30 22:52:58.467340 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.467311    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8gs\" (UniqueName: \"kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.467526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.467361    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.467526 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.467397    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.497644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.497611    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:58.497878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.497847    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:52:58.497972 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.497912    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:58.568720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.568656    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8gs\" (UniqueName: \"kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.568720 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.568721    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.568959 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.568760    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.569450 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.569426    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.571044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.571024    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.631073 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.631039    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8gs\" (UniqueName: \"kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs\") pod \"maas-api-revoke-sub-046e1343-5tk9b\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:58.711990 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:58.711960    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:52:59.087242 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:59.087205    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"]
Oct 30 22:52:59.088466 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:52:59.088442    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb35869_5272_4ec5_9d2e_efe9e7dbc3ac.slice/crio-dc9b6283b7d72a049a124c102378fa45942a2fa71af0c31b6bd93253ed08cf2e WatchSource:0}: Error finding container dc9b6283b7d72a049a124c102378fa45942a2fa71af0c31b6bd93253ed08cf2e: Status 404 returned error can't find the container with id dc9b6283b7d72a049a124c102378fa45942a2fa71af0c31b6bd93253ed08cf2e
Oct 30 22:52:59.670363 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:59.670330    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b" event={"ID":"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac","Type":"ContainerStarted","Data":"eaa6c955e26b1fc2799d1464cd408ceb5c7365594ca8fabfe8cb82f6fa0dca18"}
Oct 30 22:52:59.670816 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:59.670380    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b" event={"ID":"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac","Type":"ContainerStarted","Data":"dc9b6283b7d72a049a124c102378fa45942a2fa71af0c31b6bd93253ed08cf2e"}
Oct 30 22:52:59.786220 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:52:59.786164    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b" podStartSLOduration=1.786150621 podStartE2EDuration="1.786150621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:59.784183109 +0000 UTC m=+3110.633696083" watchObservedRunningTime="2026-10-30 22:52:59.786150621 +0000 UTC m=+3110.635663662"
Oct 30 22:53:00.686310 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:00.686272    5028 generic.go:356] "Generic (PLEG): container finished" podID="1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" containerID="eaa6c955e26b1fc2799d1464cd408ceb5c7365594ca8fabfe8cb82f6fa0dca18" exitCode=0
Oct 30 22:53:00.686780 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:00.686321    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b" event={"ID":"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac","Type":"ContainerDied","Data":"eaa6c955e26b1fc2799d1464cd408ceb5c7365594ca8fabfe8cb82f6fa0dca18"}
Oct 30 22:53:01.837782 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.837752    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:53:01.898533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.898499    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs\" (UniqueName: \"kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs\") pod \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") "
Oct 30 22:53:01.898533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.898538    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca\") pod \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") "
Oct 30 22:53:01.898812 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.898574    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca\") pod \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\" (UID: \"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac\") "
Oct 30 22:53:01.899038 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.899006    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca" pod "1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" (UID: "1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:01.900862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.900838    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca" pod "1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" (UID: "1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:01.900862 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.900850    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs" pod "1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" (UID: "1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac"). InnerVolumeSpecName "kube-api-access-rl8gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:01.999926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.999841    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:01.999926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.999874    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl8gs\" (UniqueName: \"kubernetes.io/projected/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-kube-api-access-rl8gs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:01.999926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:01.999884    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:02.700944 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:02.700914    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"
Oct 30 22:53:02.701147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:02.700955    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b" event={"ID":"1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac","Type":"ContainerDied","Data":"dc9b6283b7d72a049a124c102378fa45942a2fa71af0c31b6bd93253ed08cf2e"}
Oct 30 22:53:02.701147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:02.701000    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9b6283b7d72a049a124c102378fa45942a2fa71af0c31b6bd93253ed08cf2e"
Oct 30 22:53:09.183225 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.183189    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"]
Oct 30 22:53:09.183655 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.183641    5028 cpu_manager.go:414] "containerMap: removing container" podUID="1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" containerName="revoke-keys"
Oct 30 22:53:09.183723 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.183655    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" containerName="revoke-keys"
Oct 30 22:53:09.183773 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.183751    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" containerName="revoke-keys"
Oct 30 22:53:09.186936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.186918    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.274058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.274023    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.274212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.274172    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdtl\" (UniqueName: \"kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.274276 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.274256    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.287398 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.287374    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:09.287740 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.287679    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:09.287940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.287817    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:09.287940 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.287900    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:53:09.290825 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.290806    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"]
Oct 30 22:53:09.375478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.375440    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.375707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.375628    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdtl\" (UniqueName: \"kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.375707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.375671    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.424612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.424580    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:09.424612 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.424598    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:09.426183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.426152    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.428516 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.428487    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.482077 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.482045    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:09.485838 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.485815    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdtl\" (UniqueName: \"kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl\") pod \"maas-api-revoke-sub-d3fba056-jpcxw\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.549303 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.549269    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:53:09.551374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.551358    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:09.925247 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:09.925214    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"]
Oct 30 22:53:09.926431 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:53:09.926406    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c950952_5aa7_4133_8b70_634f64e71ee1.slice/crio-9be43472477e513391c92dddf94e2511fc59de0ecddab0238ec9a432569d771b WatchSource:0}: Error finding container 9be43472477e513391c92dddf94e2511fc59de0ecddab0238ec9a432569d771b: Status 404 returned error can't find the container with id 9be43472477e513391c92dddf94e2511fc59de0ecddab0238ec9a432569d771b
Oct 30 22:53:10.763982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:10.763945    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw" event={"ID":"0c950952-5aa7-4133-8b70-634f64e71ee1","Type":"ContainerStarted","Data":"ec97fb4155d7377c5da0a8c25b2dd84c6d8bce74a29ac6ae6e17c4bfa5606eed"}
Oct 30 22:53:10.764543 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:10.764037    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw" event={"ID":"0c950952-5aa7-4133-8b70-634f64e71ee1","Type":"ContainerStarted","Data":"9be43472477e513391c92dddf94e2511fc59de0ecddab0238ec9a432569d771b"}
Oct 30 22:53:10.877549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:10.877496    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw" podStartSLOduration=1.8774813510000001 podStartE2EDuration="1.877481351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:10.875751383 +0000 UTC m=+3121.725264359" watchObservedRunningTime="2026-10-30 22:53:10.877481351 +0000 UTC m=+3121.726994323"
Oct 30 22:53:11.776553 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:11.776493    5028 generic.go:356] "Generic (PLEG): container finished" podID="0c950952-5aa7-4133-8b70-634f64e71ee1" containerID="ec97fb4155d7377c5da0a8c25b2dd84c6d8bce74a29ac6ae6e17c4bfa5606eed" exitCode=0
Oct 30 22:53:11.777105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:11.776583    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw" event={"ID":"0c950952-5aa7-4133-8b70-634f64e71ee1","Type":"ContainerDied","Data":"ec97fb4155d7377c5da0a8c25b2dd84c6d8bce74a29ac6ae6e17c4bfa5606eed"}
Oct 30 22:53:13.789764 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.783895    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:13.815876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.815841    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw" event={"ID":"0c950952-5aa7-4133-8b70-634f64e71ee1","Type":"ContainerDied","Data":"9be43472477e513391c92dddf94e2511fc59de0ecddab0238ec9a432569d771b"}
Oct 30 22:53:13.816160 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.816139    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be43472477e513391c92dddf94e2511fc59de0ecddab0238ec9a432569d771b"
Oct 30 22:53:13.816349 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.816335    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"
Oct 30 22:53:13.856587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.855711    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca\") pod \"0c950952-5aa7-4133-8b70-634f64e71ee1\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") "
Oct 30 22:53:13.856587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.855773    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca\") pod \"0c950952-5aa7-4133-8b70-634f64e71ee1\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") "
Oct 30 22:53:13.856587 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.855848    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl\" (UniqueName: \"kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl\") pod \"0c950952-5aa7-4133-8b70-634f64e71ee1\" (UID: \"0c950952-5aa7-4133-8b70-634f64e71ee1\") "
Oct 30 22:53:13.857618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.857592    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca" pod "0c950952-5aa7-4133-8b70-634f64e71ee1" (UID: "0c950952-5aa7-4133-8b70-634f64e71ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:13.955721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.953244    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl" pod "0c950952-5aa7-4133-8b70-634f64e71ee1" (UID: "0c950952-5aa7-4133-8b70-634f64e71ee1"). InnerVolumeSpecName "kube-api-access-ppdtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:13.955721 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.953377    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca" pod "0c950952-5aa7-4133-8b70-634f64e71ee1" (UID: "0c950952-5aa7-4133-8b70-634f64e71ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:13.980710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.966860    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:13.980710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.966901    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c950952-5aa7-4133-8b70-634f64e71ee1-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:13.980710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:13.966922    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppdtl\" (UniqueName: \"kubernetes.io/projected/0c950952-5aa7-4133-8b70-634f64e71ee1-kube-api-access-ppdtl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:17.698332 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:17.698291    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:19.377607 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:19.377571    5028 scope.go:122] "RemoveContainer" containerID="e2daae6f4c1ffa6444be914dede4a873186a524c8acc9abadbe5a5ebb3bbdf1d"
Oct 30 22:53:19.387408 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:19.387384    5028 scope.go:122] "RemoveContainer" containerID="800a77ddd894fc66f358bdcc5e4d0411431e7fdafad8f8c31e1a90b301bd8f9d"
Oct 30 22:53:19.400563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:19.400538    5028 scope.go:122] "RemoveContainer" containerID="09018c129d8099b5edde0d821f2fc65942d7f4cd46f527ab6a0448c845294ce3"
Oct 30 22:53:19.409464 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:19.409446    5028 scope.go:122] "RemoveContainer" containerID="e05b87115fa899dc8bd4f285f7c10a999324515ee5ece0d74e1bd61ea0f97fe9"
Oct 30 22:53:28.457047 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:53:28.456968    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:53:44.006033 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.005989    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"]
Oct 30 22:53:44.006743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.006457    5028 cpu_manager.go:414] "containerMap: removing container" podUID="0c950952-5aa7-4133-8b70-634f64e71ee1" containerName="revoke-keys"
Oct 30 22:53:44.006743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.006468    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c950952-5aa7-4133-8b70-634f64e71ee1" containerName="revoke-keys"
Oct 30 22:53:44.006743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.006551    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c950952-5aa7-4133-8b70-634f64e71ee1" containerName="revoke-keys"
Oct 30 22:53:44.010178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.010152    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.066527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.066492    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"]
Oct 30 22:53:44.109019 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.108986    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:44.109211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.109057    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:44.109211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.109097    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:53:44.109211 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.109110    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:44.152379 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.152351    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwc4\" (UniqueName: \"kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.152552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.152427    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.152552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.152515    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.253701 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.253663    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.253876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.253779    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwc4\" (UniqueName: \"kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.253876 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.253823    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.254315 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.254293    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.256127 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.256084    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.312711 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.312659    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwc4\" (UniqueName: \"kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4\") pod \"maas-api-revoke-sub-ae594979-ts5lj\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.323480 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.323454    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:44.696625 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:44.696600    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"]
Oct 30 22:53:44.697952 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:53:44.697924    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8131d2_9e45_4e38_bf9b_2a7770df2ac2.slice/crio-49d7bde61e7519bb7fb267502d9f7a3efc0ed1709b6e7c93d8c0978ac3823dcb WatchSource:0}: Error finding container 49d7bde61e7519bb7fb267502d9f7a3efc0ed1709b6e7c93d8c0978ac3823dcb: Status 404 returned error can't find the container with id 49d7bde61e7519bb7fb267502d9f7a3efc0ed1709b6e7c93d8c0978ac3823dcb
Oct 30 22:53:45.086307 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:45.086261    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj" event={"ID":"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2","Type":"ContainerStarted","Data":"458b3084202df6ce218cc0647ff92fd6a2849ae6b305383761f279732c6ce89a"}
Oct 30 22:53:45.086776 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:45.086318    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj" event={"ID":"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2","Type":"ContainerStarted","Data":"49d7bde61e7519bb7fb267502d9f7a3efc0ed1709b6e7c93d8c0978ac3823dcb"}
Oct 30 22:53:45.199139 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:45.199080    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj" podStartSLOduration=2.199065049 podStartE2EDuration="2.199065049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:45.198050028 +0000 UTC m=+3156.047563004" watchObservedRunningTime="2026-10-30 22:53:45.199065049 +0000 UTC m=+3156.048578029"
Oct 30 22:53:46.095046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:46.095010    5028 generic.go:356] "Generic (PLEG): container finished" podID="8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" containerID="458b3084202df6ce218cc0647ff92fd6a2849ae6b305383761f279732c6ce89a" exitCode=0
Oct 30 22:53:46.095454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:46.095092    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj" event={"ID":"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2","Type":"ContainerDied","Data":"458b3084202df6ce218cc0647ff92fd6a2849ae6b305383761f279732c6ce89a"}
Oct 30 22:53:47.243855 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.243827    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:47.385796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.385719    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca\") pod \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") "
Oct 30 22:53:47.385796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.385779    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4\" (UniqueName: \"kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4\") pod \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") "
Oct 30 22:53:47.386058 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.385838    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca\") pod \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\" (UID: \"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2\") "
Oct 30 22:53:47.386329 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.386294    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca" pod "8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" (UID: "8e8131d2-9e45-4e38-bf9b-2a7770df2ac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:47.388558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.388527    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca" pod "8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" (UID: "8e8131d2-9e45-4e38-bf9b-2a7770df2ac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:47.388665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.388532    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4" pod "8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" (UID: "8e8131d2-9e45-4e38-bf9b-2a7770df2ac2"). InnerVolumeSpecName "kube-api-access-6kwc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:47.487531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.487496    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:47.487531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.487525    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwc4\" (UniqueName: \"kubernetes.io/projected/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-kube-api-access-6kwc4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:47.487531 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:47.487536    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:48.109442 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:48.109407    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"
Oct 30 22:53:48.109606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:48.109440    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj" event={"ID":"8e8131d2-9e45-4e38-bf9b-2a7770df2ac2","Type":"ContainerDied","Data":"49d7bde61e7519bb7fb267502d9f7a3efc0ed1709b6e7c93d8c0978ac3823dcb"}
Oct 30 22:53:48.109606 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:53:48.109492    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d7bde61e7519bb7fb267502d9f7a3efc0ed1709b6e7c93d8c0978ac3823dcb"
Oct 30 22:53:58.435219 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:53:58.435181    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:54:04.678614 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:04.678534    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:05.123864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:05.123820    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:54:05.124216 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:05.124163    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:54:12.098459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.098425    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"]
Oct 30 22:54:12.098946 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.098930    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" containerName="revoke-keys"
Oct 30 22:54:12.099002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.098947    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" containerName="revoke-keys"
Oct 30 22:54:12.099043 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.099031    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" containerName="revoke-keys"
Oct 30 22:54:12.102401 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.102378    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.158722 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.158009    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"]
Oct 30 22:54:12.171045 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.171010    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bbn\" (UniqueName: \"kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.171255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.171065    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.171255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.171201    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.201431 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.201395    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:12.201618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.201438    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:12.201618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.201448    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:54:12.201618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.201500    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:12.271991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.271963    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.272208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.272146    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bbn\" (UniqueName: \"kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.272208 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.272200    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.272632 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.272611    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.274611 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.274589    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.331201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.331174    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bbn\" (UniqueName: \"kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn\") pod \"maas-api-revoke-sub-a4aef7d7-fpnk5\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.415218 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.415141    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:12.783463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:12.783432    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"]
Oct 30 22:54:12.784538 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:54:12.784512    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ad80a1_beb2_4fac_afef_3cd377a3b490.slice/crio-58a62a71c522224aa6c9c21be7e6fe307012b19e91886d9ca483bac987d06d9d WatchSource:0}: Error finding container 58a62a71c522224aa6c9c21be7e6fe307012b19e91886d9ca483bac987d06d9d: Status 404 returned error can't find the container with id 58a62a71c522224aa6c9c21be7e6fe307012b19e91886d9ca483bac987d06d9d
Oct 30 22:54:13.277520 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:13.277478    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5" event={"ID":"c1ad80a1-beb2-4fac-afef-3cd377a3b490","Type":"ContainerStarted","Data":"f9706dee5bfa49ae119cbcd58d6db338f17c061d17bb75a64d47252626c72aba"}
Oct 30 22:54:13.277956 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:13.277537    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5" event={"ID":"c1ad80a1-beb2-4fac-afef-3cd377a3b490","Type":"ContainerStarted","Data":"58a62a71c522224aa6c9c21be7e6fe307012b19e91886d9ca483bac987d06d9d"}
Oct 30 22:54:13.389936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:13.389842    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5" podStartSLOduration=1.389828426 podStartE2EDuration="1.389828426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:13.388454542 +0000 UTC m=+3184.237967516" watchObservedRunningTime="2026-10-30 22:54:13.389828426 +0000 UTC m=+3184.239341399"
Oct 30 22:54:14.289032 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:14.288996    5028 generic.go:356] "Generic (PLEG): container finished" podID="c1ad80a1-beb2-4fac-afef-3cd377a3b490" containerID="f9706dee5bfa49ae119cbcd58d6db338f17c061d17bb75a64d47252626c72aba" exitCode=0
Oct 30 22:54:14.289485 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:14.289074    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5" event={"ID":"c1ad80a1-beb2-4fac-afef-3cd377a3b490","Type":"ContainerDied","Data":"f9706dee5bfa49ae119cbcd58d6db338f17c061d17bb75a64d47252626c72aba"}
Oct 30 22:54:15.438355 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.438330    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:15.504212 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.504178    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca\") pod \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") "
Oct 30 22:54:15.504397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.504266    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn\" (UniqueName: \"kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn\") pod \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") "
Oct 30 22:54:15.504397 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.504305    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca\") pod \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\" (UID: \"c1ad80a1-beb2-4fac-afef-3cd377a3b490\") "
Oct 30 22:54:15.504704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.504666    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca" pod "c1ad80a1-beb2-4fac-afef-3cd377a3b490" (UID: "c1ad80a1-beb2-4fac-afef-3cd377a3b490"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:15.506441 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.506416    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca" pod "c1ad80a1-beb2-4fac-afef-3cd377a3b490" (UID: "c1ad80a1-beb2-4fac-afef-3cd377a3b490"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:15.506545 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.506421    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn" pod "c1ad80a1-beb2-4fac-afef-3cd377a3b490" (UID: "c1ad80a1-beb2-4fac-afef-3cd377a3b490"). InnerVolumeSpecName "kube-api-access-22bbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:15.605250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.605171    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.605250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.605200    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ad80a1-beb2-4fac-afef-3cd377a3b490-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.605250 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:15.605212    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bbn\" (UniqueName: \"kubernetes.io/projected/c1ad80a1-beb2-4fac-afef-3cd377a3b490-kube-api-access-22bbn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:16.304317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:16.304286    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"
Oct 30 22:54:16.304508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:16.304317    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5" event={"ID":"c1ad80a1-beb2-4fac-afef-3cd377a3b490","Type":"ContainerDied","Data":"58a62a71c522224aa6c9c21be7e6fe307012b19e91886d9ca483bac987d06d9d"}
Oct 30 22:54:16.304508 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:16.304368    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a62a71c522224aa6c9c21be7e6fe307012b19e91886d9ca483bac987d06d9d"
Oct 30 22:54:18.043926 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:18.043893    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:19.444783 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:19.444701    5028 scope.go:122] "RemoveContainer" containerID="34423e18a2eca2b9010fb4bd63c8e52d776b41a0c65dc64f942ddff827153824"
Oct 30 22:54:19.454499 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:19.454475    5028 scope.go:122] "RemoveContainer" containerID="b4d35d13d302bd49b262bb91e6f30cc5ddeacf10223e49054352e238ec780a4c"
Oct 30 22:54:19.463052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:19.463033    5028 scope.go:122] "RemoveContainer" containerID="64abe0da11e6003ad4b74a804956cb0fcbc30ed0e87099e893c433bfbca8b09c"
Oct 30 22:54:19.471580 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:19.471558    5028 scope.go:122] "RemoveContainer" containerID="4deca31b7339d0edf4ffcc2fbe1f27a427f0b134e83f8124e5bd2f1d34bfd162"
Oct 30 22:54:28.435497 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:54:28.435453    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:54:35.123868 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:35.123791    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wgcfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:54:35.124425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:35.123865    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wgcfm" podUID="b3fe4656-786a-40d9-977c-83d2bbf357cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:54:39.055105 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:39.055069    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"]
Oct 30 22:54:39.102793 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:39.102750    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ad84f8a-7m2mn"]
Oct 30 22:54:39.290335 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:39.290291    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dd0114-0c8c-4bab-83fe-e6df2c18175c" path="/var/lib/kubelet/pods/67dd0114-0c8c-4bab-83fe-e6df2c18175c/volumes"
Oct 30 22:54:45.909781 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:45.909742    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"]
Oct 30 22:54:45.910386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:45.910363    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c1ad80a1-beb2-4fac-afef-3cd377a3b490" containerName="revoke-keys"
Oct 30 22:54:45.910476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:45.910388    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ad80a1-beb2-4fac-afef-3cd377a3b490" containerName="revoke-keys"
Oct 30 22:54:45.910573 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:45.910549    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ad80a1-beb2-4fac-afef-3cd377a3b490" containerName="revoke-keys"
Oct 30 22:54:45.913919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:45.913898    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:45.965110 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:45.965080    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:46.012041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.012000    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:46.012041 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.012001    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:46.012289 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.012136    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:54:46.012330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.012296    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.012388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.012372    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.012496 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.012478    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvc7\" (UniqueName: \"kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.014935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.014912    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"]
Oct 30 22:54:46.114215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.114180    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.114402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.114290    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvc7\" (UniqueName: \"kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.114402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.114338    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.115031 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.115006    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.116744 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.116724    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.174955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.174875    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvc7\" (UniqueName: \"kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7\") pod \"maas-api-revoke-sub-79805ca9-pzqp8\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.226933 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.226899    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:46.605917 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:46.605541    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"]
Oct 30 22:54:47.549271 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:47.549233    5028 generic.go:356] "Generic (PLEG): container finished" podID="a9019bb1-70fb-4a84-ac40-beb1116a11ff" containerID="0e00e3d79fb050c974d51a9b313d5612703cb402e0935ced5586f43d8235c5f6" exitCode=0
Oct 30 22:54:47.549660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:47.549297    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8" event={"ID":"a9019bb1-70fb-4a84-ac40-beb1116a11ff","Type":"ContainerDied","Data":"0e00e3d79fb050c974d51a9b313d5612703cb402e0935ced5586f43d8235c5f6"}
Oct 30 22:54:47.549660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:47.549345    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8" event={"ID":"a9019bb1-70fb-4a84-ac40-beb1116a11ff","Type":"ContainerStarted","Data":"a0477245600bbd1ab801a161dbe94fd8e6e7d43f2b231c76c56da8c01e40b8b6"}
Oct 30 22:54:48.700420 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.700393    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:48.741787 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.741750    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca\") pod \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") "
Oct 30 22:54:48.741951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.741809    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca\") pod \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") "
Oct 30 22:54:48.741951 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.741839    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7\" (UniqueName: \"kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7\") pod \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\" (UID: \"a9019bb1-70fb-4a84-ac40-beb1116a11ff\") "
Oct 30 22:54:48.742150 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.742126    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca" pod "a9019bb1-70fb-4a84-ac40-beb1116a11ff" (UID: "a9019bb1-70fb-4a84-ac40-beb1116a11ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:48.742312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.742287    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:48.744089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.744065    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca" pod "a9019bb1-70fb-4a84-ac40-beb1116a11ff" (UID: "a9019bb1-70fb-4a84-ac40-beb1116a11ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:48.744172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.744089    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7" pod "a9019bb1-70fb-4a84-ac40-beb1116a11ff" (UID: "a9019bb1-70fb-4a84-ac40-beb1116a11ff"). InnerVolumeSpecName "kube-api-access-rvvc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:48.843214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.843133    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9019bb1-70fb-4a84-ac40-beb1116a11ff-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:48.843214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:48.843163    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvc7\" (UniqueName: \"kubernetes.io/projected/a9019bb1-70fb-4a84-ac40-beb1116a11ff-kube-api-access-rvvc7\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.562939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:49.562858    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"
Oct 30 22:54:49.562939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:49.562892    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8" event={"ID":"a9019bb1-70fb-4a84-ac40-beb1116a11ff","Type":"ContainerDied","Data":"a0477245600bbd1ab801a161dbe94fd8e6e7d43f2b231c76c56da8c01e40b8b6"}
Oct 30 22:54:49.562939 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:54:49.562937    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0477245600bbd1ab801a161dbe94fd8e6e7d43f2b231c76c56da8c01e40b8b6"
Oct 30 22:54:58.435365 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:54:58.435325    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:55:09.801373 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.801332    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"]
Oct 30 22:55:09.801900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.801812    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a9019bb1-70fb-4a84-ac40-beb1116a11ff" containerName="revoke-keys"
Oct 30 22:55:09.801900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.801824    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9019bb1-70fb-4a84-ac40-beb1116a11ff" containerName="revoke-keys"
Oct 30 22:55:09.801900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.801886    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9019bb1-70fb-4a84-ac40-beb1116a11ff" containerName="revoke-keys"
Oct 30 22:55:09.805654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.805618    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.893523 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.893485    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.893704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.893528    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.893704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.893616    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkmk\" (UniqueName: \"kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.903761 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.903731    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:09.903950 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.903927    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:55:09.904000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.903941    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:09.904044 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.903993    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:09.906914 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.906894    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"]
Oct 30 22:55:09.994454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.994409    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.994454 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.994455    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.994766 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.994508    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkmk\" (UniqueName: \"kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.995156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.995124    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:09.996919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:09.996898    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:10.054107 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:10.054028    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkmk\" (UniqueName: \"kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk\") pod \"maas-api-revoke-sub-0b7e8edb-gnf9s\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:10.122421 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:10.122389    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:10.506651 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:10.506623    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"]
Oct 30 22:55:10.508044 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:55:10.508006    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bd27a3_0a1d_4cc5_b548_705f26a7c3b1.slice/crio-5cbccca76e55ff48773c9431b5eb1a7870919d74d3d5eeb43745449bca1090a7 WatchSource:0}: Error finding container 5cbccca76e55ff48773c9431b5eb1a7870919d74d3d5eeb43745449bca1090a7: Status 404 returned error can't find the container with id 5cbccca76e55ff48773c9431b5eb1a7870919d74d3d5eeb43745449bca1090a7
Oct 30 22:55:10.741106 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:10.741075    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s" event={"ID":"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1","Type":"ContainerStarted","Data":"5cbccca76e55ff48773c9431b5eb1a7870919d74d3d5eeb43745449bca1090a7"}
Oct 30 22:55:11.748158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:11.748125    5028 generic.go:356] "Generic (PLEG): container finished" podID="68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" containerID="6e3aa96ccc9dcfd739ff98edd55fb675f817d32873bad7c335f49784620a1eb0" exitCode=0
Oct 30 22:55:11.748598 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:11.748192    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s" event={"ID":"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1","Type":"ContainerDied","Data":"6e3aa96ccc9dcfd739ff98edd55fb675f817d32873bad7c335f49784620a1eb0"}
Oct 30 22:55:12.894935 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:12.894907    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:13.023648 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.023576    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk\" (UniqueName: \"kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk\") pod \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") "
Oct 30 22:55:13.023852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.023658    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca\") pod \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") "
Oct 30 22:55:13.023852 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.023714    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca\") pod \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\" (UID: \"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1\") "
Oct 30 22:55:13.024103 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.024078    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca" pod "68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" (UID: "68bd27a3-0a1d-4cc5-b548-705f26a7c3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:13.025943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.025916    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca" pod "68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" (UID: "68bd27a3-0a1d-4cc5-b548-705f26a7c3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:13.026040 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.025975    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk" pod "68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" (UID: "68bd27a3-0a1d-4cc5-b548-705f26a7c3b1"). InnerVolumeSpecName "kube-api-access-flkmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:13.124858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.124804    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flkmk\" (UniqueName: \"kubernetes.io/projected/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-kube-api-access-flkmk\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:13.124858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.124854    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:13.124858 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.124866    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:13.760970 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.760944    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"
Oct 30 22:55:13.761152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.760980    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s" event={"ID":"68bd27a3-0a1d-4cc5-b548-705f26a7c3b1","Type":"ContainerDied","Data":"5cbccca76e55ff48773c9431b5eb1a7870919d74d3d5eeb43745449bca1090a7"}
Oct 30 22:55:13.761152 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:13.761049    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbccca76e55ff48773c9431b5eb1a7870919d74d3d5eeb43745449bca1090a7"
Oct 30 22:55:18.384871 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:18.384834    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:19.506563 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:19.506528    5028 scope.go:122] "RemoveContainer" containerID="8cca0ce12d410e45237d96d6ed51b97e2d67119552de8baff415d06445a83fe2"
Oct 30 22:55:19.516265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:19.516244    5028 scope.go:122] "RemoveContainer" containerID="6daa61f174776a1111808acfbde2da65db6ba2345048aa60dbd22dfc7c0ba4c0"
Oct 30 22:55:21.243004 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.242963    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"]
Oct 30 22:55:21.243429 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.243416    5028 cpu_manager.go:414] "containerMap: removing container" podUID="68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" containerName="revoke-keys"
Oct 30 22:55:21.243488 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.243430    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" containerName="revoke-keys"
Oct 30 22:55:21.243533 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.243524    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" containerName="revoke-keys"
Oct 30 22:55:21.246796 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.246776    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.297936 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.297904    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.298143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.297960    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvrb\" (UniqueName: \"kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.298143 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.298062    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.305558 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.305527    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"]
Oct 30 22:55:21.348390 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.348354    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:21.348665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.348642    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:55:21.348665 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.348680    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:21.348851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.348683    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:21.399111 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.399079    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvrb\" (UniqueName: \"kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.399277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.399126    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.399277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.399221    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.399924 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.399898    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.401518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.401497    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.458348 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.458312    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvrb\" (UniqueName: \"kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb\") pod \"maas-api-revoke-sub-3d4156f1-9wbvc\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.560443 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.560343    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:21.929756 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:21.929731    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"]
Oct 30 22:55:21.930827 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:55:21.930791    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb030bbe1_20e5_4961_91e9_0834c3267c58.slice/crio-d3575cab34c7bde5a6ca41389bae6fd408ad144eac07a705eb8af63ee29818bc WatchSource:0}: Error finding container d3575cab34c7bde5a6ca41389bae6fd408ad144eac07a705eb8af63ee29818bc: Status 404 returned error can't find the container with id d3575cab34c7bde5a6ca41389bae6fd408ad144eac07a705eb8af63ee29818bc
Oct 30 22:55:22.830357 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:22.830317    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc" event={"ID":"b030bbe1-20e5-4961-91e9-0834c3267c58","Type":"ContainerStarted","Data":"506e3c8e2e0772eba41ef0d4e8bb2c848022a6bd88d2a1532030d7f88c16f2cf"}
Oct 30 22:55:22.830785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:22.830413    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc" event={"ID":"b030bbe1-20e5-4961-91e9-0834c3267c58","Type":"ContainerStarted","Data":"d3575cab34c7bde5a6ca41389bae6fd408ad144eac07a705eb8af63ee29818bc"}
Oct 30 22:55:22.944196 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:22.944142    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc" podStartSLOduration=1.944127708 podStartE2EDuration="1.944127708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:22.941531502 +0000 UTC m=+3253.791044478" watchObservedRunningTime="2026-10-30 22:55:22.944127708 +0000 UTC m=+3253.793640686"
Oct 30 22:55:23.883252 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:23.883212    5028 generic.go:356] "Generic (PLEG): container finished" podID="b030bbe1-20e5-4961-91e9-0834c3267c58" containerID="506e3c8e2e0772eba41ef0d4e8bb2c848022a6bd88d2a1532030d7f88c16f2cf" exitCode=0
Oct 30 22:55:23.883955 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:23.883928    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc" event={"ID":"b030bbe1-20e5-4961-91e9-0834c3267c58","Type":"ContainerDied","Data":"506e3c8e2e0772eba41ef0d4e8bb2c848022a6bd88d2a1532030d7f88c16f2cf"}
Oct 30 22:55:25.224524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.224498    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:25.269245 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.269208    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca\") pod \"b030bbe1-20e5-4961-91e9-0834c3267c58\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") "
Oct 30 22:55:25.269425 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.269262    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb\" (UniqueName: \"kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb\") pod \"b030bbe1-20e5-4961-91e9-0834c3267c58\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") "
Oct 30 22:55:25.269664 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.269638    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca" pod "b030bbe1-20e5-4961-91e9-0834c3267c58" (UID: "b030bbe1-20e5-4961-91e9-0834c3267c58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:25.271659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.271635    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb" pod "b030bbe1-20e5-4961-91e9-0834c3267c58" (UID: "b030bbe1-20e5-4961-91e9-0834c3267c58"). InnerVolumeSpecName "kube-api-access-tfvrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:25.370227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.370200    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca\") pod \"b030bbe1-20e5-4961-91e9-0834c3267c58\" (UID: \"b030bbe1-20e5-4961-91e9-0834c3267c58\") "
Oct 30 22:55:25.370472 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.370453    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvrb\" (UniqueName: \"kubernetes.io/projected/b030bbe1-20e5-4961-91e9-0834c3267c58-kube-api-access-tfvrb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:25.370564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.370473    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:25.372415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.372381    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca" pod "b030bbe1-20e5-4961-91e9-0834c3267c58" (UID: "b030bbe1-20e5-4961-91e9-0834c3267c58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:25.471785 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.471748    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b030bbe1-20e5-4961-91e9-0834c3267c58-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:25.904824 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.904743    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"
Oct 30 22:55:25.904981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.904746    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc" event={"ID":"b030bbe1-20e5-4961-91e9-0834c3267c58","Type":"ContainerDied","Data":"d3575cab34c7bde5a6ca41389bae6fd408ad144eac07a705eb8af63ee29818bc"}
Oct 30 22:55:25.904981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:25.904866    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3575cab34c7bde5a6ca41389bae6fd408ad144eac07a705eb8af63ee29818bc"
Oct 30 22:55:28.435399 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:55:28.435359    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:55:31.469788 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:31.469752    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:31.628054 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:31.628011    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:33.079274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:33.079239    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"]
Oct 30 22:55:33.126194 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:33.126160    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pb758"]
Oct 30 22:55:33.290359 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:33.290322    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a259025-215f-4c31-a544-ed87b7b08de1" path="/var/lib/kubelet/pods/4a259025-215f-4c31-a544-ed87b7b08de1/volumes"
Oct 30 22:55:55.107214 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.107132    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"]
Oct 30 22:55:55.153795 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.153761    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9wbvc"]
Oct 30 22:55:55.224052 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.224010    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"]
Oct 30 22:55:55.224493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.224478    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b030bbe1-20e5-4961-91e9-0834c3267c58" containerName="revoke-keys"
Oct 30 22:55:55.224544 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.224494    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b030bbe1-20e5-4961-91e9-0834c3267c58" containerName="revoke-keys"
Oct 30 22:55:55.224589 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.224580    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b030bbe1-20e5-4961-91e9-0834c3267c58" containerName="revoke-keys"
Oct 30 22:55:55.227986 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.227969    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.279130 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.279100    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:55.279386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.279245    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:55.279386 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.279294    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:55:55.279604 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.279310    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:55.283046 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.283020    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"]
Oct 30 22:55:55.291704 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.291665    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b030bbe1-20e5-4961-91e9-0834c3267c58" path="/var/lib/kubelet/pods/b030bbe1-20e5-4961-91e9-0834c3267c58/volumes"
Oct 30 22:55:55.318026 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.317993    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbfr\" (UniqueName: \"kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.318172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.318033    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.318172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.318152    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.419063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.418979    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbfr\" (UniqueName: \"kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.419063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.419028    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.419317 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.419145    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.419890 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.419867    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.421557 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.421534    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.478174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.478137    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbfr\" (UniqueName: \"kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr\") pod \"maas-api-revoke-sub-3d4156f1-8xbkv\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.541681 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.541647    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:55.913259 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:55.913220    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"]
Oct 30 22:55:55.914698 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:55:55.914657    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07b811e_a215_4721_b0e4_c636c56372bf.slice/crio-cd8653c324b2b7d11236f81d23d4473aea6241c06812fc52e524d4457537f182 WatchSource:0}: Error finding container cd8653c324b2b7d11236f81d23d4473aea6241c06812fc52e524d4457537f182: Status 404 returned error can't find the container with id cd8653c324b2b7d11236f81d23d4473aea6241c06812fc52e524d4457537f182
Oct 30 22:55:56.127393 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:56.127174    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv" event={"ID":"c07b811e-a215-4721-b0e4-c636c56372bf","Type":"ContainerStarted","Data":"cd8653c324b2b7d11236f81d23d4473aea6241c06812fc52e524d4457537f182"}
Oct 30 22:55:57.135330 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:57.135243    5028 generic.go:356] "Generic (PLEG): container finished" podID="c07b811e-a215-4721-b0e4-c636c56372bf" containerID="a4417f1cfde97a90588595434d482c3a4c6097ec4048cb0d1e2e0cbf25976aad" exitCode=0
Oct 30 22:55:57.135743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:57.135327    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv" event={"ID":"c07b811e-a215-4721-b0e4-c636c56372bf","Type":"ContainerDied","Data":"a4417f1cfde97a90588595434d482c3a4c6097ec4048cb0d1e2e0cbf25976aad"}
Oct 30 22:55:58.283518 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.283488    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:58.338892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.338859    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca\") pod \"c07b811e-a215-4721-b0e4-c636c56372bf\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") "
Oct 30 22:55:58.339082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.338930    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca\") pod \"c07b811e-a215-4721-b0e4-c636c56372bf\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") "
Oct 30 22:55:58.339082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.339055    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr\" (UniqueName: \"kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr\") pod \"c07b811e-a215-4721-b0e4-c636c56372bf\" (UID: \"c07b811e-a215-4721-b0e4-c636c56372bf\") "
Oct 30 22:55:58.339284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.339256    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca" pod "c07b811e-a215-4721-b0e4-c636c56372bf" (UID: "c07b811e-a215-4721-b0e4-c636c56372bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:58.339415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.339400    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:58.341109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.341082    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca" pod "c07b811e-a215-4721-b0e4-c636c56372bf" (UID: "c07b811e-a215-4721-b0e4-c636c56372bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:58.341228 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.341210    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr" pod "c07b811e-a215-4721-b0e4-c636c56372bf" (UID: "c07b811e-a215-4721-b0e4-c636c56372bf"). InnerVolumeSpecName "kube-api-access-vbbfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:58.436044 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:55:58.435946    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:55:58.440650 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.440630    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbfr\" (UniqueName: \"kubernetes.io/projected/c07b811e-a215-4721-b0e4-c636c56372bf-kube-api-access-vbbfr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:58.440755 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:58.440652    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b811e-a215-4721-b0e4-c636c56372bf-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:59.148981 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:59.148950    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"
Oct 30 22:55:59.149172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:59.148986    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv" event={"ID":"c07b811e-a215-4721-b0e4-c636c56372bf","Type":"ContainerDied","Data":"cd8653c324b2b7d11236f81d23d4473aea6241c06812fc52e524d4457537f182"}
Oct 30 22:55:59.149172 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:55:59.149034    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8653c324b2b7d11236f81d23d4473aea6241c06812fc52e524d4457537f182"
Oct 30 22:56:10.564374 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:10.564338    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:56:10.603861 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:10.603835    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 22:56:18.717089 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:18.717053    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:19.542879 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:19.542846    5028 scope.go:122] "RemoveContainer" containerID="cdc45cf73c4ba1f7f4ae62093aa0e9ecd44ac16f827deb3d9baa4029c66aa49e"
Oct 30 22:56:28.442539 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:56:28.442495    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:56:30.383988 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:30.383955    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:43.051524 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:43.051489    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"]
Oct 30 22:56:43.098255 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:43.098221    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ttlzn"]
Oct 30 22:56:43.290025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:43.289990    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417b9032-3182-4762-bd8b-0ac547ba5d68" path="/var/lib/kubelet/pods/417b9032-3182-4762-bd8b-0ac547ba5d68/volumes"
Oct 30 22:56:45.827510 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.827474    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"]
Oct 30 22:56:45.828002 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.827987    5028 cpu_manager.go:414] "containerMap: removing container" podUID="c07b811e-a215-4721-b0e4-c636c56372bf" containerName="revoke-keys"
Oct 30 22:56:45.828057 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.828003    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07b811e-a215-4721-b0e4-c636c56372bf" containerName="revoke-keys"
Oct 30 22:56:45.828113 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.828103    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07b811e-a215-4721-b0e4-c636c56372bf" containerName="revoke-keys"
Oct 30 22:56:45.831385 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.831369    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.868082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.868046    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6nj\" (UniqueName: \"kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.868082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.868087    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.868308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.868231    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.887446 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.887406    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"]
Oct 30 22:56:45.930982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.930947    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:45.930982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.930966    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:45.930982 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.930971    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:45.931347 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.931069    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:56:45.969717 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.969670    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.969893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.969755    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6nj\" (UniqueName: \"kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.969893 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.969776    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.970459 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.970430    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:45.972109 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:45.972084    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:46.030036 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:46.029998    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6nj\" (UniqueName: \"kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj\") pod \"maas-api-revoke-sub-e2fc9c5b-f44xx\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:46.144187 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:46.144108    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:46.513234 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:46.513205    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"]
Oct 30 22:56:46.513829 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:56:46.513805    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a097bc2_bd8a_413f_ab1e_4053630dbd8e.slice/crio-b64e7b53263dbd67d76d97f29982872313cff96b435f3b3abddd4e4fdb554d1e WatchSource:0}: Error finding container b64e7b53263dbd67d76d97f29982872313cff96b435f3b3abddd4e4fdb554d1e: Status 404 returned error can't find the container with id b64e7b53263dbd67d76d97f29982872313cff96b435f3b3abddd4e4fdb554d1e
Oct 30 22:56:47.477572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:47.477529    5028 generic.go:356] "Generic (PLEG): container finished" podID="7a097bc2-bd8a-413f-ab1e-4053630dbd8e" containerID="8762e12a39400216bb3c086d933a7e2cd06d6c8810972697369b680b7d680df1" exitCode=0
Oct 30 22:56:47.477997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:47.477608    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx" event={"ID":"7a097bc2-bd8a-413f-ab1e-4053630dbd8e","Type":"ContainerDied","Data":"8762e12a39400216bb3c086d933a7e2cd06d6c8810972697369b680b7d680df1"}
Oct 30 22:56:47.477997 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:47.477658    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx" event={"ID":"7a097bc2-bd8a-413f-ab1e-4053630dbd8e","Type":"ContainerStarted","Data":"b64e7b53263dbd67d76d97f29982872313cff96b435f3b3abddd4e4fdb554d1e"}
Oct 30 22:56:48.620539 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.620510    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:48.695461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.695427    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj\" (UniqueName: \"kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj\") pod \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") "
Oct 30 22:56:48.695661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.695469    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca\") pod \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") "
Oct 30 22:56:48.695661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.695502    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca\") pod \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\" (UID: \"7a097bc2-bd8a-413f-ab1e-4053630dbd8e\") "
Oct 30 22:56:48.695998 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.695968    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca" pod "7a097bc2-bd8a-413f-ab1e-4053630dbd8e" (UID: "7a097bc2-bd8a-413f-ab1e-4053630dbd8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:48.697603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.697576    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj" pod "7a097bc2-bd8a-413f-ab1e-4053630dbd8e" (UID: "7a097bc2-bd8a-413f-ab1e-4053630dbd8e"). InnerVolumeSpecName "kube-api-access-mm6nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:48.697603 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.697588    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca" pod "7a097bc2-bd8a-413f-ab1e-4053630dbd8e" (UID: "7a097bc2-bd8a-413f-ab1e-4053630dbd8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:48.797246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.797162    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:48.797246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.797194    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:48.797246 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:48.797205    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6nj\" (UniqueName: \"kubernetes.io/projected/7a097bc2-bd8a-413f-ab1e-4053630dbd8e-kube-api-access-mm6nj\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:49.491215 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:49.491186    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"
Oct 30 22:56:49.491388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:49.491221    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx" event={"ID":"7a097bc2-bd8a-413f-ab1e-4053630dbd8e","Type":"ContainerDied","Data":"b64e7b53263dbd67d76d97f29982872313cff96b435f3b3abddd4e4fdb554d1e"}
Oct 30 22:56:49.491388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:49.491274    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b64e7b53263dbd67d76d97f29982872313cff96b435f3b3abddd4e4fdb554d1e"
Oct 30 22:56:56.553380 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.553341    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"]
Oct 30 22:56:56.553851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.553807    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7a097bc2-bd8a-413f-ab1e-4053630dbd8e" containerName="revoke-keys"
Oct 30 22:56:56.553851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.553822    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a097bc2-bd8a-413f-ab1e-4053630dbd8e" containerName="revoke-keys"
Oct 30 22:56:56.553960 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.553926    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a097bc2-bd8a-413f-ab1e-4053630dbd8e" containerName="revoke-keys"
Oct 30 22:56:56.557370 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.557351    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.612626 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.612589    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"]
Oct 30 22:56:56.655144 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.655116    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:56.655343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.655183    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:56.655343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.655184    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:56.655343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.655208    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:56:56.672581 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.672556    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.672742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.672589    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzjs\" (UniqueName: \"kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.672742 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.672618    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.774017 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.773977    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.774229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.774028    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzjs\" (UniqueName: \"kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.774229 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.774074    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.774833 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.774807    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.776493 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.776467    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.832837 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.832760    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzjs\" (UniqueName: \"kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs\") pod \"maas-api-revoke-sub-5a29efb3-t9htp\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:56.870753 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:56.870722    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:57.239809 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:57.239777    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"]
Oct 30 22:56:57.240665 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:56:57.240641    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc1a7c8_e5c8_4569_9c1d_235457f6c1c6.slice/crio-da3165e3a15a16333614ea649c54b86aa60e8ac396ad31524d958146805fdf15 WatchSource:0}: Error finding container da3165e3a15a16333614ea649c54b86aa60e8ac396ad31524d958146805fdf15: Status 404 returned error can't find the container with id da3165e3a15a16333614ea649c54b86aa60e8ac396ad31524d958146805fdf15
Oct 30 22:56:57.540511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:57.540473    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp" event={"ID":"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6","Type":"ContainerStarted","Data":"11e31960a903522244084c2877fb238af00d0a06330d6a9942b590d80fc3c71a"}
Oct 30 22:56:57.540667 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:57.540531    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp" event={"ID":"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6","Type":"ContainerStarted","Data":"da3165e3a15a16333614ea649c54b86aa60e8ac396ad31524d958146805fdf15"}
Oct 30 22:56:57.652592 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:57.652527    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp" podStartSLOduration=1.652513321 podStartE2EDuration="1.652513321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:57.651298088 +0000 UTC m=+3348.500811077" watchObservedRunningTime="2026-10-30 22:56:57.652513321 +0000 UTC m=+3348.502026293"
Oct 30 22:56:58.436468 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:56:58.436387    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:56:58.548774 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:58.548738    5028 generic.go:356] "Generic (PLEG): container finished" podID="cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" containerID="11e31960a903522244084c2877fb238af00d0a06330d6a9942b590d80fc3c71a" exitCode=0
Oct 30 22:56:58.549025 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:58.548825    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp" event={"ID":"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6","Type":"ContainerDied","Data":"11e31960a903522244084c2877fb238af00d0a06330d6a9942b590d80fc3c71a"}
Oct 30 22:56:59.782145 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.781794    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:56:59.924644 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.924543    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca\") pod \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") "
Oct 30 22:56:59.924885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.924645    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca\") pod \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") "
Oct 30 22:56:59.924885 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.924800    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs\" (UniqueName: \"kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs\") pod \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\" (UID: \"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6\") "
Oct 30 22:56:59.925808 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.925760    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca" pod "cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" (UID: "cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:59.928140 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.928097    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs" pod "cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" (UID: "cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6"). InnerVolumeSpecName "kube-api-access-4jzjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:59.929896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:56:59.929860    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca" pod "cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" (UID: "cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:00.026900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:00.026815    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzjs\" (UniqueName: \"kubernetes.io/projected/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-kube-api-access-4jzjs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:00.026900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:00.026855    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:00.026900 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:00.026872    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:00.614532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:00.613729    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp" event={"ID":"cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6","Type":"ContainerDied","Data":"da3165e3a15a16333614ea649c54b86aa60e8ac396ad31524d958146805fdf15"}
Oct 30 22:57:00.614532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:00.613822    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da3165e3a15a16333614ea649c54b86aa60e8ac396ad31524d958146805fdf15"
Oct 30 22:57:00.614532 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:00.613944    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"
Oct 30 22:57:13.580803 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.580765    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"]
Oct 30 22:57:13.628597 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.628552    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8xbkv"]
Oct 30 22:57:13.698489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.698451    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"]
Oct 30 22:57:13.699010 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.698992    5028 cpu_manager.go:414] "containerMap: removing container" podUID="cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" containerName="revoke-keys"
Oct 30 22:57:13.699131 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.699011    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" containerName="revoke-keys"
Oct 30 22:57:13.699201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.699158    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" containerName="revoke-keys"
Oct 30 22:57:13.702300 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.702280    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.736745 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.736679    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.736877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.736807    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6m5\" (UniqueName: \"kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.736877 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.736870    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.755710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.755491    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:13.755710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.755586    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:13.755710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.755663    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:13.755710 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.755680    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:57:13.759439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.759417    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"]
Oct 30 22:57:13.837969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.837878    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6m5\" (UniqueName: \"kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.837969 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.837938    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.838231 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.838004    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.838631 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.838610    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.840572 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.840543    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:13.897201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:13.897164    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6m5\" (UniqueName: \"kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5\") pod \"maas-api-revoke-sub-3d4156f1-gbnnd\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:14.015698 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:14.015662    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:14.049037 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:14.048995    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"]
Oct 30 22:57:14.051736 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:14.051706    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9xkns"]
Oct 30 22:57:14.387174 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:14.387144    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"]
Oct 30 22:57:14.387991 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:57:14.387967    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9e0c01_3d44_438a_9365_883f19d43e18.slice/crio-e2c3a51ecbca0a8fe3595efcbe6a74f7efb50fc58f5d7c84e6bae71e3fc2f27d WatchSource:0}: Error finding container e2c3a51ecbca0a8fe3595efcbe6a74f7efb50fc58f5d7c84e6bae71e3fc2f27d: Status 404 returned error can't find the container with id e2c3a51ecbca0a8fe3595efcbe6a74f7efb50fc58f5d7c84e6bae71e3fc2f27d
Oct 30 22:57:14.729284 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:14.729249    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd" event={"ID":"ca9e0c01-3d44-438a-9365-883f19d43e18","Type":"ContainerStarted","Data":"e2c3a51ecbca0a8fe3595efcbe6a74f7efb50fc58f5d7c84e6bae71e3fc2f27d"}
Oct 30 22:57:15.289624 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:15.289592    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19088ec-f220-4a4f-b5dc-347d277e2127" path="/var/lib/kubelet/pods/a19088ec-f220-4a4f-b5dc-347d277e2127/volumes"
Oct 30 22:57:15.290707 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:15.290672    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07b811e-a215-4721-b0e4-c636c56372bf" path="/var/lib/kubelet/pods/c07b811e-a215-4721-b0e4-c636c56372bf/volumes"
Oct 30 22:57:15.737466 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:15.737430    5028 generic.go:356] "Generic (PLEG): container finished" podID="ca9e0c01-3d44-438a-9365-883f19d43e18" containerID="91d4fdb34eb729b330bd3e3c3bcf5bc9e4369e683a897cf79084961cc46cedca" exitCode=0
Oct 30 22:57:15.737892 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:15.737521    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd" event={"ID":"ca9e0c01-3d44-438a-9365-883f19d43e18","Type":"ContainerDied","Data":"91d4fdb34eb729b330bd3e3c3bcf5bc9e4369e683a897cf79084961cc46cedca"}
Oct 30 22:57:16.893422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.893399    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:16.968233 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.968204    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca\") pod \"ca9e0c01-3d44-438a-9365-883f19d43e18\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") "
Oct 30 22:57:16.968439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.968284    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca\") pod \"ca9e0c01-3d44-438a-9365-883f19d43e18\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") "
Oct 30 22:57:16.968439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.968318    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5\" (UniqueName: \"kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5\") pod \"ca9e0c01-3d44-438a-9365-883f19d43e18\" (UID: \"ca9e0c01-3d44-438a-9365-883f19d43e18\") "
Oct 30 22:57:16.968737 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.968677    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca" pod "ca9e0c01-3d44-438a-9365-883f19d43e18" (UID: "ca9e0c01-3d44-438a-9365-883f19d43e18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:16.970527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.970502    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca" pod "ca9e0c01-3d44-438a-9365-883f19d43e18" (UID: "ca9e0c01-3d44-438a-9365-883f19d43e18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:16.970527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:16.970510    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5" pod "ca9e0c01-3d44-438a-9365-883f19d43e18" (UID: "ca9e0c01-3d44-438a-9365-883f19d43e18"). InnerVolumeSpecName "kube-api-access-zl6m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:17.069647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:17.069559    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:17.069647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:17.069595    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e0c01-3d44-438a-9365-883f19d43e18-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:17.069647 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:17.069616    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl6m5\" (UniqueName: \"kubernetes.io/projected/ca9e0c01-3d44-438a-9365-883f19d43e18-kube-api-access-zl6m5\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:17.751535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:17.751494    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd" event={"ID":"ca9e0c01-3d44-438a-9365-883f19d43e18","Type":"ContainerDied","Data":"e2c3a51ecbca0a8fe3595efcbe6a74f7efb50fc58f5d7c84e6bae71e3fc2f27d"}
Oct 30 22:57:17.751535 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:17.751517    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"
Oct 30 22:57:17.751768 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:17.751543    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2c3a51ecbca0a8fe3595efcbe6a74f7efb50fc58f5d7c84e6bae71e3fc2f27d"
Oct 30 22:57:19.125219 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:19.125183    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:19.309752 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:19.309650    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:19.566503 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:19.566469    5028 scope.go:122] "RemoveContainer" containerID="92b8ee8007126c4e6773ec2cdcf19aff27b9e9f296cd40aa53dad4938da34367"
Oct 30 22:57:27.059201 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:27.059170    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"]
Oct 30 22:57:27.106402 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:27.106369    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-c9lbl"]
Oct 30 22:57:27.290132 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:27.290103    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f624e487-f8b8-4e3a-a885-0235614d3e68" path="/var/lib/kubelet/pods/f624e487-f8b8-4e3a-a885-0235614d3e68/volumes"
Oct 30 22:57:28.435234 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:57:28.435200    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:57:44.851929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:44.851892    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:45.487463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:45.487360    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:46.109501 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:46.109459    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:46.322206 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:46.322172    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:50.323059 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:50.323022    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:50.485640 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:50.485563    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:50.645564 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:50.645471    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:53.591277 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.591238    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:53.744555 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.744509    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:53.891101 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.891015    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"]
Oct 30 22:57:53.891495 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.891481    5028 cpu_manager.go:414] "containerMap: removing container" podUID="ca9e0c01-3d44-438a-9365-883f19d43e18" containerName="revoke-keys"
Oct 30 22:57:53.891550 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.891495    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9e0c01-3d44-438a-9365-883f19d43e18" containerName="revoke-keys"
Oct 30 22:57:53.891594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.891566    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9e0c01-3d44-438a-9365-883f19d43e18" containerName="revoke-keys"
Oct 30 22:57:53.895137 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.895119    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:53.954878 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.954829    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"]
Oct 30 22:57:53.994517 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.994485    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dcc58b\""
Oct 30 22:57:53.994728 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.994572    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-471e3a52-dockercfg-zmshq\""
Oct 30 22:57:53.994991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.994965    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78ld\" (UniqueName: \"kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld\") pod \"payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:53.995147 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:53.995127    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:54.096489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:54.096450    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78ld\" (UniqueName: \"kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld\") pod \"payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:54.096672 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:54.096578    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:54.097334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:54.097308    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:54.155743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:54.155663    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78ld\" (UniqueName: \"kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld\") pod \"payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:54.208929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:54.208893    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:54.579463 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:54.579434    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"]
Oct 30 22:57:54.580429 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:57:54.580406    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba1e1d6_3f84_4ba9_a262_96368d5cd474.slice/crio-9767bf2985272a230f4d218e45e5e07e5d135f36ada74bbad12d18c181dfd44c WatchSource:0}: Error finding container 9767bf2985272a230f4d218e45e5e07e5d135f36ada74bbad12d18c181dfd44c: Status 404 returned error can't find the container with id 9767bf2985272a230f4d218e45e5e07e5d135f36ada74bbad12d18c181dfd44c
Oct 30 22:57:55.016461 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:55.016424    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" event={"ID":"8ba1e1d6-3f84-4ba9-a262-96368d5cd474","Type":"ContainerStarted","Data":"49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a"}
Oct 30 22:57:55.016904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:55.016479    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" event={"ID":"8ba1e1d6-3f84-4ba9-a262-96368d5cd474","Type":"ContainerStarted","Data":"9767bf2985272a230f4d218e45e5e07e5d135f36ada74bbad12d18c181dfd44c"}
Oct 30 22:57:55.016904 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:55.016501    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:57:55.129993 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:55.129940    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" podStartSLOduration=2.129924858 podStartE2EDuration="2.129924858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:55.128118296 +0000 UTC m=+3405.977631271" watchObservedRunningTime="2026-10-30 22:57:55.129924858 +0000 UTC m=+3405.979437837"
Oct 30 22:57:56.288974 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:56.288942    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:56.441527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:57:56.441492    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:58.435811 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:57:58.435776    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:58:00.957161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:00.957129    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:02.048476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:02.048433    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"]
Oct 30 22:58:02.094715 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:02.094666    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5tk9b"]
Oct 30 22:58:03.289794 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:03.289763    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac" path="/var/lib/kubelet/pods/1eb35869-5272-4ec5-9d2e-efe9e7dbc3ac/volumes"
Oct 30 22:58:03.492515 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:03.492482    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:05.681818 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:05.681784    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:06.025659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:06.025629    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:58:10.284757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:10.284716    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:14.046542 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:14.046504    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"]
Oct 30 22:58:14.093528 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:14.093495    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jpcxw"]
Oct 30 22:58:15.065296 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:15.065250    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:15.223821 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:15.223777    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:15.291497 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:15.291463    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c950952-5aa7-4133-8b70-634f64e71ee1" path="/var/lib/kubelet/pods/0c950952-5aa7-4133-8b70-634f64e71ee1/volumes"
Oct 30 22:58:15.808677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:15.808638    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:15.974600 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:15.974571    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:16.127494 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:16.127411    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:18.748334 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:18.748301    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:19.412843 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:19.412811    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:19.590929 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:19.590895    5028 scope.go:122] "RemoveContainer" containerID="22b5cdb1cfdae8f29f8631d4f01103c46268716320be0b1baaae00ce0f8a98fa"
Oct 30 22:58:19.600566 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:19.600546    5028 scope.go:122] "RemoveContainer" containerID="1950c900803297cac6dd049a7e344c1923f63bf166298bbf85574b4af48287a4"
Oct 30 22:58:19.768476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:19.768441    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:24.349171 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:24.349130    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:28.436560 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:58:28.436514    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:58:28.846082 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:28.846031    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:33.359197 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:33.359163    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:37.846099 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:37.846066    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:42.441891 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:42.441858    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:46.945830 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:46.945797    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:48.049312 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:48.049275    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"]
Oct 30 22:58:48.096323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:48.096286    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ts5lj"]
Oct 30 22:58:49.298161 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:49.298079    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8131d2-9e45-4e38-bf9b-2a7770df2ac2" path="/var/lib/kubelet/pods/8e8131d2-9e45-4e38-bf9b-2a7770df2ac2/volumes"
Oct 30 22:58:51.578834 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:51.578795    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:56.167778 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:58:56.167739    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:58.438975 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:58:58.438927    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:59:00.752899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:00.752868    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:03.427973 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.427935    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"]
Oct 30 22:59:03.431674 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.431652    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.483290 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.483251    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:59:03.487301 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.487266    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"]
Oct 30 22:59:03.502020 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.501988    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.502185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.502040    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.502185 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.502165    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ls46\" (UniqueName: \"kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.530487 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.530458    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:59:03.530660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.530493    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:59:03.530660 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.530560    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:59:03.603184 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.603154    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ls46\" (UniqueName: \"kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.603388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.603263    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.603388 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.603325    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.604080 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.604051    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.605670 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.605653    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.664416 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.664379    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ls46\" (UniqueName: \"kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46\") pod \"maas-api-revoke-sub-3999c1b9-bng87\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:03.744943 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:03.744913    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:04.123475 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:04.123445    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"]
Oct 30 22:59:04.123898 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:59:04.123876    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45ad654_117f_463d_a2e0_04935931ea9e.slice/crio-701ad1ff9edace8f0e6f2a3298995cefddda2b93897c0e92af2b4a9de15ad5f4 WatchSource:0}: Error finding container 701ad1ff9edace8f0e6f2a3298995cefddda2b93897c0e92af2b4a9de15ad5f4: Status 404 returned error can't find the container with id 701ad1ff9edace8f0e6f2a3298995cefddda2b93897c0e92af2b4a9de15ad5f4
Oct 30 22:59:04.460325 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:04.460290    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87" event={"ID":"e45ad654-117f-463d-a2e0-04935931ea9e","Type":"ContainerStarted","Data":"81eb722f4469144a2bf2dde13965e07879720c2440823e6c4ba7c6e93982988b"}
Oct 30 22:59:04.460817 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:04.460344    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87" event={"ID":"e45ad654-117f-463d-a2e0-04935931ea9e","Type":"ContainerStarted","Data":"701ad1ff9edace8f0e6f2a3298995cefddda2b93897c0e92af2b4a9de15ad5f4"}
Oct 30 22:59:04.572633 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:04.572575    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87" podStartSLOduration=1.572561627 podStartE2EDuration="1.572561627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:04.571633264 +0000 UTC m=+3475.421146236" watchObservedRunningTime="2026-10-30 22:59:04.572561627 +0000 UTC m=+3475.422074594"
Oct 30 22:59:05.341519 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:05.341435    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:05.468122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:05.468083    5028 generic.go:356] "Generic (PLEG): container finished" podID="e45ad654-117f-463d-a2e0-04935931ea9e" containerID="81eb722f4469144a2bf2dde13965e07879720c2440823e6c4ba7c6e93982988b" exitCode=0
Oct 30 22:59:05.468561 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:05.468179    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87" event={"ID":"e45ad654-117f-463d-a2e0-04935931ea9e","Type":"ContainerDied","Data":"81eb722f4469144a2bf2dde13965e07879720c2440823e6c4ba7c6e93982988b"}
Oct 30 22:59:06.616492 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.616465    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:06.627352 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.627329    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca\") pod \"e45ad654-117f-463d-a2e0-04935931ea9e\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") "
Oct 30 22:59:06.627489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.627425    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca\") pod \"e45ad654-117f-463d-a2e0-04935931ea9e\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") "
Oct 30 22:59:06.627489 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.627459    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46\" (UniqueName: \"kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46\") pod \"e45ad654-117f-463d-a2e0-04935931ea9e\" (UID: \"e45ad654-117f-463d-a2e0-04935931ea9e\") "
Oct 30 22:59:06.627654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.627631    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca" pod "e45ad654-117f-463d-a2e0-04935931ea9e" (UID: "e45ad654-117f-463d-a2e0-04935931ea9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:06.627743 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.627716    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:06.629504 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.629477    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca" pod "e45ad654-117f-463d-a2e0-04935931ea9e" (UID: "e45ad654-117f-463d-a2e0-04935931ea9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:06.629609 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.629556    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46" pod "e45ad654-117f-463d-a2e0-04935931ea9e" (UID: "e45ad654-117f-463d-a2e0-04935931ea9e"). InnerVolumeSpecName "kube-api-access-7ls46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:06.728351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.728264    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45ad654-117f-463d-a2e0-04935931ea9e-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:06.728351 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:06.728294    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ls46\" (UniqueName: \"kubernetes.io/projected/e45ad654-117f-463d-a2e0-04935931ea9e-kube-api-access-7ls46\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:07.482323 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:07.482288    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"
Oct 30 22:59:07.482527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:07.482324    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87" event={"ID":"e45ad654-117f-463d-a2e0-04935931ea9e","Type":"ContainerDied","Data":"701ad1ff9edace8f0e6f2a3298995cefddda2b93897c0e92af2b4a9de15ad5f4"}
Oct 30 22:59:07.482527 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:07.482399    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="701ad1ff9edace8f0e6f2a3298995cefddda2b93897c0e92af2b4a9de15ad5f4"
Oct 30 22:59:09.941525 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:09.941488    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:14.453747 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:14.453674    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:15.080415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.080382    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:15.589308 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.589268    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"]
Oct 30 22:59:15.590014 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.589992    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e45ad654-117f-463d-a2e0-04935931ea9e" containerName="revoke-keys"
Oct 30 22:59:15.590119 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.590016    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45ad654-117f-463d-a2e0-04935931ea9e" containerName="revoke-keys"
Oct 30 22:59:15.590203 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.590190    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45ad654-117f-463d-a2e0-04935931ea9e" containerName="revoke-keys"
Oct 30 22:59:15.593540 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.593517    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.645784 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.645751    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:59:15.692749 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.692716    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:59:15.692915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.692717    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:59:15.692915 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.692774    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:59:15.695757 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.695736    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"]
Oct 30 22:59:15.712417 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.712387    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.712552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.712454    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.712552 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.712534    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhflw\" (UniqueName: \"kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.814008 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.813975    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.814227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.814015    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhflw\" (UniqueName: \"kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.814227 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.814104    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.814884 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.814861    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.816549 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.816523    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.876000 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.875928    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhflw\" (UniqueName: \"kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:15.907654 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:15.907622    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:16.048851 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:16.048810    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"]
Oct 30 22:59:16.052511 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:16.052483    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fpnk5"]
Oct 30 22:59:16.283007 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:16.282974    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"]
Oct 30 22:59:16.283630 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:59:16.283602    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e179ba_d2cc_4ddf_883c_ec060c22ee7a.slice/crio-549630e9769f0856ec192382ff66571bd0d501cb344061554d778f533d948c2b WatchSource:0}: Error finding container 549630e9769f0856ec192382ff66571bd0d501cb344061554d778f533d948c2b: Status 404 returned error can't find the container with id 549630e9769f0856ec192382ff66571bd0d501cb344061554d778f533d948c2b
Oct 30 22:59:16.540183 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:16.540115    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx" event={"ID":"31e179ba-d2cc-4ddf-883c-ec060c22ee7a","Type":"ContainerStarted","Data":"e8e7f3a5a46579f0b085b97bef796015937b56bec95d6374bea8363e11c2598d"}
Oct 30 22:59:16.540293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:16.540183    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx" event={"ID":"31e179ba-d2cc-4ddf-883c-ec060c22ee7a","Type":"ContainerStarted","Data":"549630e9769f0856ec192382ff66571bd0d501cb344061554d778f533d948c2b"}
Oct 30 22:59:16.653864 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:16.653798    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx" podStartSLOduration=1.653782796 podStartE2EDuration="1.653782796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:16.650730751 +0000 UTC m=+3487.500243751" watchObservedRunningTime="2026-10-30 22:59:16.653782796 +0000 UTC m=+3487.503295769"
Oct 30 22:59:17.289653 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:17.289607    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ad80a1-beb2-4fac-afef-3cd377a3b490" path="/var/lib/kubelet/pods/c1ad80a1-beb2-4fac-afef-3cd377a3b490/volumes"
Oct 30 22:59:17.549055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:17.549020    5028 generic.go:356] "Generic (PLEG): container finished" podID="31e179ba-d2cc-4ddf-883c-ec060c22ee7a" containerID="e8e7f3a5a46579f0b085b97bef796015937b56bec95d6374bea8363e11c2598d" exitCode=0
Oct 30 22:59:17.549265 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:17.549063    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx" event={"ID":"31e179ba-d2cc-4ddf-883c-ec060c22ee7a","Type":"ContainerDied","Data":"e8e7f3a5a46579f0b085b97bef796015937b56bec95d6374bea8363e11c2598d"}
Oct 30 22:59:18.741090 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.741051    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:18.848618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.848531    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw\" (UniqueName: \"kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw\") pod \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") "
Oct 30 22:59:18.848618 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.848580    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca\") pod \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") "
Oct 30 22:59:18.848911 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.848635    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca\") pod \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\" (UID: \"31e179ba-d2cc-4ddf-883c-ec060c22ee7a\") "
Oct 30 22:59:18.849149 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.849113    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca" pod "31e179ba-d2cc-4ddf-883c-ec060c22ee7a" (UID: "31e179ba-d2cc-4ddf-883c-ec060c22ee7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:18.850859 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.850831    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca" pod "31e179ba-d2cc-4ddf-883c-ec060c22ee7a" (UID: "31e179ba-d2cc-4ddf-883c-ec060c22ee7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:18.850991 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.850963    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw" pod "31e179ba-d2cc-4ddf-883c-ec060c22ee7a" (UID: "31e179ba-d2cc-4ddf-883c-ec060c22ee7a"). InnerVolumeSpecName "kube-api-access-rhflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:18.950538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.950504    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:18.950538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.950533    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhflw\" (UniqueName: \"kubernetes.io/projected/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-kube-api-access-rhflw\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:18.950538 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:18.950543    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31e179ba-d2cc-4ddf-883c-ec060c22ee7a-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:19.361894 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.361850    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:19.563506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.563408    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx" event={"ID":"31e179ba-d2cc-4ddf-883c-ec060c22ee7a","Type":"ContainerDied","Data":"549630e9769f0856ec192382ff66571bd0d501cb344061554d778f533d948c2b"}
Oct 30 22:59:19.563506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.563456    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="549630e9769f0856ec192382ff66571bd0d501cb344061554d778f533d948c2b"
Oct 30 22:59:19.563506 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.563461    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"
Oct 30 22:59:19.629065 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.629033    5028 scope.go:122] "RemoveContainer" containerID="348381be5759955917e4ffad256a19fbae17d1dc9c7d6213c490228bfb6606a8"
Oct 30 22:59:19.641967 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.641842    5028 scope.go:122] "RemoveContainer" containerID="ec97fb4155d7377c5da0a8c25b2dd84c6d8bce74a29ac6ae6e17c4bfa5606eed"
Oct 30 22:59:19.661024 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.660664    5028 scope.go:122] "RemoveContainer" containerID="eaa6c955e26b1fc2799d1464cd408ceb5c7365594ca8fabfe8cb82f6fa0dca18"
Oct 30 22:59:19.771754 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:19.771459    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:20.131505 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:20.131462    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:23.947122 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:23.947086    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:27.115575 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:27.115541    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"]
Oct 30 22:59:27.373343 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:27.373251    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:27.527086 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:27.527044    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:28.012156 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:28.012111    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:28.461586 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:59:28.461551    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 22:59:31.514092 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:31.514054    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:31.729919 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:31.729879    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:32.039158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:32.039129    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"]
Oct 30 22:59:32.086695 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:32.086644    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dcc58b-55mvx"]
Oct 30 22:59:32.116142 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:32.116101    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerName="payload-pre-processing" containerID="cri-o://49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a" gracePeriod=25
Oct 30 22:59:32.504439 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:32.504409    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:33.289554 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:33.289521    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e179ba-d2cc-4ddf-883c-ec060c22ee7a" path="/var/lib/kubelet/pods/31e179ba-d2cc-4ddf-883c-ec060c22ee7a/volumes"
Oct 30 22:59:36.024406 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:36.024365    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:59:38.760163 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:38.760094    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:38.936896 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:38.936864    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:39.169311 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:39.169224    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:40.699478 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.699441    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"]
Oct 30 22:59:40.746063 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.746029    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gbnnd"]
Oct 30 22:59:40.817178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.817141    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"]
Oct 30 22:59:40.817615 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.817602    5028 cpu_manager.go:414] "containerMap: removing container" podUID="31e179ba-d2cc-4ddf-883c-ec060c22ee7a" containerName="revoke-keys"
Oct 30 22:59:40.817661 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.817617    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e179ba-d2cc-4ddf-883c-ec060c22ee7a" containerName="revoke-keys"
Oct 30 22:59:40.817738 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.817730    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e179ba-d2cc-4ddf-883c-ec060c22ee7a" containerName="revoke-keys"
Oct 30 22:59:40.822594 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.822574    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:40.873178 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.873122    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:59:40.873395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.873238    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 22:59:40.873395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.873241    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:59:40.873395 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.873244    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:59:40.875470 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.875442    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"]
Oct 30 22:59:40.983456 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.983422    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:40.983635 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.983558    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:40.983677 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:40.983635    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqn4n\" (UniqueName: \"kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.084415 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.084375    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqn4n\" (UniqueName: \"kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.084608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.084430    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.084608 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.084493    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.085293 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.085261    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.087180 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.087114    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.144392 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.144356    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqn4n\" (UniqueName: \"kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n\") pod \"maas-api-revoke-sub-3d4156f1-n8b8g\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.290338 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.290260    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9e0c01-3d44-438a-9365-883f19d43e18" path="/var/lib/kubelet/pods/ca9e0c01-3d44-438a-9365-883f19d43e18/volumes"
Oct 30 22:59:41.435683 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.435643    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:41.791513 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.791454    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:41.807712 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:41.807666    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"]
Oct 30 22:59:41.808516 ip-10-0-129-125 kubenswrapper[5028]: W1030 22:59:41.808490    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d6a054_2fc5_4ac3_9d24_5a255831b93d.slice/crio-450d08d80f551125f36a96b511610af55c4af22b0f833bfb5eaf8037d9c61847 WatchSource:0}: Error finding container 450d08d80f551125f36a96b511610af55c4af22b0f833bfb5eaf8037d9c61847: Status 404 returned error can't find the container with id 450d08d80f551125f36a96b511610af55c4af22b0f833bfb5eaf8037d9c61847
Oct 30 22:59:42.730267 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:42.730232    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g" event={"ID":"18d6a054-2fc5-4ac3-9d24-5a255831b93d","Type":"ContainerStarted","Data":"f3995497b1fa52e99bef22dc876a8894475900407892d811f8d9cdef6f98c129"}
Oct 30 22:59:42.730422 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:42.730289    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g" event={"ID":"18d6a054-2fc5-4ac3-9d24-5a255831b93d","Type":"ContainerStarted","Data":"450d08d80f551125f36a96b511610af55c4af22b0f833bfb5eaf8037d9c61847"}
Oct 30 22:59:43.738153 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.738116    5028 generic.go:356] "Generic (PLEG): container finished" podID="18d6a054-2fc5-4ac3-9d24-5a255831b93d" containerID="f3995497b1fa52e99bef22dc876a8894475900407892d811f8d9cdef6f98c129" exitCode=0
Oct 30 22:59:43.738583 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.738206    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g" event={"ID":"18d6a054-2fc5-4ac3-9d24-5a255831b93d","Type":"ContainerDied","Data":"f3995497b1fa52e99bef22dc876a8894475900407892d811f8d9cdef6f98c129"}
Oct 30 22:59:43.886158 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.886129    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:43.906476 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.906443    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca\") pod \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") "
Oct 30 22:59:43.906613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.906497    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca\") pod \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") "
Oct 30 22:59:43.906613 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.906532    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n\" (UniqueName: \"kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n\") pod \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\" (UID: \"18d6a054-2fc5-4ac3-9d24-5a255831b93d\") "
Oct 30 22:59:43.906888 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.906867    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca" pod "18d6a054-2fc5-4ac3-9d24-5a255831b93d" (UID: "18d6a054-2fc5-4ac3-9d24-5a255831b93d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:43.908682 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.908659    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca" pod "18d6a054-2fc5-4ac3-9d24-5a255831b93d" (UID: "18d6a054-2fc5-4ac3-9d24-5a255831b93d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:59:43.908805 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:43.908681    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n" pod "18d6a054-2fc5-4ac3-9d24-5a255831b93d" (UID: "18d6a054-2fc5-4ac3-9d24-5a255831b93d"). InnerVolumeSpecName "kube-api-access-nqn4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:44.007659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:44.007587    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:44.007659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:44.007616    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d6a054-2fc5-4ac3-9d24-5a255831b93d-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:44.007659 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:44.007627    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqn4n\" (UniqueName: \"kubernetes.io/projected/18d6a054-2fc5-4ac3-9d24-5a255831b93d-kube-api-access-nqn4n\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:44.746022 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:44.745989    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"
Oct 30 22:59:44.746434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:44.746033    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g" event={"ID":"18d6a054-2fc5-4ac3-9d24-5a255831b93d","Type":"ContainerDied","Data":"450d08d80f551125f36a96b511610af55c4af22b0f833bfb5eaf8037d9c61847"}
Oct 30 22:59:44.746434 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:44.746083    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="450d08d80f551125f36a96b511610af55c4af22b0f833bfb5eaf8037d9c61847"
Oct 30 22:59:46.023938 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:46.023899    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:59:49.048207 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:49.048166    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"]
Oct 30 22:59:49.095274 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:49.095238    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-pzqp8"]
Oct 30 22:59:49.290297 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:49.290267    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9019bb1-70fb-4a84-ac40-beb1116a11ff" path="/var/lib/kubelet/pods/a9019bb1-70fb-4a84-ac40-beb1116a11ff/volumes"
Oct 30 22:59:52.374287 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.374223    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:59:52.488863 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.488830    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld\" (UniqueName: \"kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld\") pod \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") "
Oct 30 22:59:52.489067 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.489014    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume\") pod \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\" (UID: \"8ba1e1d6-3f84-4ba9-a262-96368d5cd474\") "
Oct 30 22:59:52.489680 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.489640    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume" pod "8ba1e1d6-3f84-4ba9-a262-96368d5cd474" (UID: "8ba1e1d6-3f84-4ba9-a262-96368d5cd474"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:59:52.491200 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.491167    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld" pod "8ba1e1d6-3f84-4ba9-a262-96368d5cd474" (UID: "8ba1e1d6-3f84-4ba9-a262-96368d5cd474"). InnerVolumeSpecName "kube-api-access-q78ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:59:52.590762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.590715    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.590762 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.590748    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q78ld\" (UniqueName: \"kubernetes.io/projected/8ba1e1d6-3f84-4ba9-a262-96368d5cd474-kube-api-access-q78ld\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:59:52.797123 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.797084    5028 generic.go:356] "Generic (PLEG): container finished" podID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerID="49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a" exitCode=0
Oct 30 22:59:52.797344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.797131    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"
Oct 30 22:59:52.797344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.797140    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" event={"ID":"8ba1e1d6-3f84-4ba9-a262-96368d5cd474","Type":"ContainerDied","Data":"49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a"}
Oct 30 22:59:52.797344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.797230    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml" event={"ID":"8ba1e1d6-3f84-4ba9-a262-96368d5cd474","Type":"ContainerDied","Data":"9767bf2985272a230f4d218e45e5e07e5d135f36ada74bbad12d18c181dfd44c"}
Oct 30 22:59:52.797344 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.797263    5028 scope.go:122] "RemoveContainer" containerID="49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a"
Oct 30 22:59:52.809716 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.809697    5028 scope.go:122] "RemoveContainer" containerID="49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a"
Oct 30 22:59:52.809983 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:59:52.809965    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a\": container with ID starting with 49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a not found: ID does not exist" containerID="49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a"
Oct 30 22:59:52.810055 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.809990    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a"} err="failed to get container status \"49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a\": rpc error: code = NotFound desc = could not find container \"49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a\": container with ID starting with 49487844959bb9d655d5d36a0321bab1ada7405b7fee89e10b8e22eb67d1852a not found: ID does not exist"
Oct 30 22:59:52.955899 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:52.955861    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"]
Oct 30 22:59:53.003135 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:53.003100    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcc58b-69b75jqmml"]
Oct 30 22:59:53.289897 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:53.289866    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" path="/var/lib/kubelet/pods/8ba1e1d6-3f84-4ba9-a262-96368d5cd474/volumes"
Oct 30 22:59:57.511283 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:57.511235    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:57.953769 ip-10-0-129-125 kubenswrapper[5028]: I1030 22:59:57.950310    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 22:59:58.435950 ip-10-0-129-125 kubenswrapper[5028]: E1030 22:59:58.435912    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:00:00.160671 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.160631    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"]
Oct 30 23:00:00.161154 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.161138    5028 cpu_manager.go:414] "containerMap: removing container" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerName="payload-pre-processing"
Oct 30 23:00:00.161201 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.161155    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerName="payload-pre-processing"
Oct 30 23:00:00.161201 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.161166    5028 cpu_manager.go:414] "containerMap: removing container" podUID="18d6a054-2fc5-4ac3-9d24-5a255831b93d" containerName="revoke-keys"
Oct 30 23:00:00.161201 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.161172    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d6a054-2fc5-4ac3-9d24-5a255831b93d" containerName="revoke-keys"
Oct 30 23:00:00.161306 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.161248    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba1e1d6-3f84-4ba9-a262-96368d5cd474" containerName="payload-pre-processing"
Oct 30 23:00:00.161306 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.161259    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d6a054-2fc5-4ac3-9d24-5a255831b93d" containerName="revoke-keys"
Oct 30 23:00:00.166006 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.165988    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.219885 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.219836    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"]
Oct 30 23:00:00.263246 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.263209    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-phnzx\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.263448 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.263368    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724dc\" (UniqueName: \"kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc\") pod \"maas-api-key-cleanup-29890020-phnzx\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.263524 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.263491    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:00:00.263524 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.263491    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:00:00.263602 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.263553    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:00:00.364833 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.364796    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724dc\" (UniqueName: \"kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc\") pod \"maas-api-key-cleanup-29890020-phnzx\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.365062 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.364893    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-phnzx\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.365590 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.365566    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-phnzx\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.424091 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.424010    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724dc\" (UniqueName: \"kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc\") pod \"maas-api-key-cleanup-29890020-phnzx\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.478777 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.478733    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:00.847803 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.847776    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"]
Oct 30 23:00:00.849456 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:00:00.849414    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f18d49_b7c0_4554_9a59_b16bb9cfa220.slice/crio-b9e8936aec46f9455d3346e500aaedadc42c2ba2929f1f161893c108ee3f30c3 WatchSource:0}: Error finding container b9e8936aec46f9455d3346e500aaedadc42c2ba2929f1f161893c108ee3f30c3: Status 404 returned error can't find the container with id b9e8936aec46f9455d3346e500aaedadc42c2ba2929f1f161893c108ee3f30c3
Oct 30 23:00:00.876425 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:00.876396    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" event={"ID":"33f18d49-b7c0-4554-9a59-b16bb9cfa220","Type":"ContainerStarted","Data":"b9e8936aec46f9455d3346e500aaedadc42c2ba2929f1f161893c108ee3f30c3"}
Oct 30 23:00:01.884440 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:01.884404    5028 generic.go:356] "Generic (PLEG): container finished" podID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerID="31632aac9e90d77a6e9ae90d7a7b5cc7cefb778717bfe789690271385104b123" exitCode=60
Oct 30 23:00:01.884924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:01.884484    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" event={"ID":"33f18d49-b7c0-4554-9a59-b16bb9cfa220","Type":"ContainerDied","Data":"31632aac9e90d77a6e9ae90d7a7b5cc7cefb778717bfe789690271385104b123"}
Oct 30 23:00:01.884924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:01.884763    5028 scope.go:122] "RemoveContainer" containerID="31632aac9e90d77a6e9ae90d7a7b5cc7cefb778717bfe789690271385104b123"
Oct 30 23:00:02.893517 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:02.893480    5028 generic.go:356] "Generic (PLEG): container finished" podID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerID="772bfbf39a8408d0cd885546237c1b1a69691e82ac99d34903ab50109e709d0a" exitCode=60
Oct 30 23:00:02.893966 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:02.893559    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" event={"ID":"33f18d49-b7c0-4554-9a59-b16bb9cfa220","Type":"ContainerDied","Data":"772bfbf39a8408d0cd885546237c1b1a69691e82ac99d34903ab50109e709d0a"}
Oct 30 23:00:02.893966 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:02.893636    5028 scope.go:122] "RemoveContainer" containerID="31632aac9e90d77a6e9ae90d7a7b5cc7cefb778717bfe789690271385104b123"
Oct 30 23:00:02.893966 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:02.893966    5028 scope.go:122] "RemoveContainer" containerID="772bfbf39a8408d0cd885546237c1b1a69691e82ac99d34903ab50109e709d0a"
Oct 30 23:00:02.894211 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:00:02.894193    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-phnzx_odh-ai-gateway-infra(33f18d49-b7c0-4554-9a59-b16bb9cfa220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220"
Oct 30 23:00:03.901367 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:03.901332    5028 scope.go:122] "RemoveContainer" containerID="772bfbf39a8408d0cd885546237c1b1a69691e82ac99d34903ab50109e709d0a"
Oct 30 23:00:03.901849 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:00:03.901538    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-phnzx_odh-ai-gateway-infra(33f18d49-b7c0-4554-9a59-b16bb9cfa220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220"
Oct 30 23:00:04.006049 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:04.006018    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:14.049315 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:14.049279    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"]
Oct 30 23:00:14.096610 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:14.096579    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gnf9s"]
Oct 30 23:00:15.289727 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.289670    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bd27a3-0a1d-4cc5-b548-705f26a7c3b1" path="/var/lib/kubelet/pods/68bd27a3-0a1d-4cc5-b548-705f26a7c3b1/volumes"
Oct 30 23:00:15.336539 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.336502    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"]
Oct 30 23:00:15.336780 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.336735    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf" podUID="5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" containerName="manager" containerID="cri-o://e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca" gracePeriod=10
Oct 30 23:00:15.613814 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.613787    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 23:00:15.715800 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.715763    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp\" (UniqueName: \"kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp\") pod \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") "
Oct 30 23:00:15.715980 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.715911    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume\") pod \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\" (UID: \"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9\") "
Oct 30 23:00:15.716291 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.716266    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume" pod "5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" (UID: "5a6194d8-8f7b-46aa-a3dc-c98949a2fba9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:00:15.717928 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.717903    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp" pod "5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" (UID: "5a6194d8-8f7b-46aa-a3dc-c98949a2fba9"). InnerVolumeSpecName "kube-api-access-pp9vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:15.817774 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.817737    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9vp\" (UniqueName: \"kubernetes.io/projected/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-kube-api-access-pp9vp\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:00:15.817774 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.817773    5028 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9-extensions-socket-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:00:15.977506 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.977472    5028 generic.go:356] "Generic (PLEG): container finished" podID="5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" containerID="e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca" exitCode=0
Oct 30 23:00:15.977721 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.977535    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"
Oct 30 23:00:15.977721 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.977575    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf" event={"ID":"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9","Type":"ContainerDied","Data":"e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca"}
Oct 30 23:00:15.977721 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.977666    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf" event={"ID":"5a6194d8-8f7b-46aa-a3dc-c98949a2fba9","Type":"ContainerDied","Data":"11f09bea5d4bf5669f25d9bbfed985126e226cb3cafddb99c86d9b12f6ca2861"}
Oct 30 23:00:15.977721 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.977714    5028 scope.go:122] "RemoveContainer" containerID="e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca"
Oct 30 23:00:15.990799 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.990780    5028 scope.go:122] "RemoveContainer" containerID="e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca"
Oct 30 23:00:15.991036 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:00:15.991019    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca\": container with ID starting with e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca not found: ID does not exist" containerID="e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca"
Oct 30 23:00:15.991116 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:15.991044    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca"} err="failed to get container status \"e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca\": rpc error: code = NotFound desc = could not find container \"e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca\": container with ID starting with e2ceb2b55528c9bdb5d90455a53037280d6a848c67489e5273430fdbd91883ca not found: ID does not exist"
Oct 30 23:00:16.138716 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:16.138663    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"]
Oct 30 23:00:16.148467 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:16.146110    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-99dlf"]
Oct 30 23:00:16.284498 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:16.284412    5028 scope.go:122] "RemoveContainer" containerID="772bfbf39a8408d0cd885546237c1b1a69691e82ac99d34903ab50109e709d0a"
Oct 30 23:00:16.986052 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:16.986015    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" event={"ID":"33f18d49-b7c0-4554-9a59-b16bb9cfa220","Type":"ContainerStarted","Data":"c09ef1e2360b688b50c1a0c39989306779165e325f9daf323d5f34d181745147"}
Oct 30 23:00:17.097577 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:17.097523    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" podStartSLOduration=17.097508278 podStartE2EDuration="17.097508278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:17.096811036 +0000 UTC m=+3547.946324014" watchObservedRunningTime="2026-10-30 23:00:17.097508278 +0000 UTC m=+3547.947021308"
Oct 30 23:00:17.290987 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:17.290906    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" path="/var/lib/kubelet/pods/5a6194d8-8f7b-46aa-a3dc-c98949a2fba9/volumes"
Oct 30 23:00:17.995313 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:17.995280    5028 generic.go:356] "Generic (PLEG): container finished" podID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerID="c09ef1e2360b688b50c1a0c39989306779165e325f9daf323d5f34d181745147" exitCode=60
Oct 30 23:00:17.995870 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:17.995355    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" event={"ID":"33f18d49-b7c0-4554-9a59-b16bb9cfa220","Type":"ContainerDied","Data":"c09ef1e2360b688b50c1a0c39989306779165e325f9daf323d5f34d181745147"}
Oct 30 23:00:17.995870 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:17.995434    5028 scope.go:122] "RemoveContainer" containerID="772bfbf39a8408d0cd885546237c1b1a69691e82ac99d34903ab50109e709d0a"
Oct 30 23:00:17.995870 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:17.995762    5028 scope.go:122] "RemoveContainer" containerID="c09ef1e2360b688b50c1a0c39989306779165e325f9daf323d5f34d181745147"
Oct 30 23:00:17.996025 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:00:17.996004    5028 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-phnzx_odh-ai-gateway-infra(33f18d49-b7c0-4554-9a59-b16bb9cfa220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220"
Oct 30 23:00:18.108083 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:18.108034    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"]
Oct 30 23:00:19.144660 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.144630    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:19.250485 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.250406    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc\" (UniqueName: \"kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc\") pod \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") "
Oct 30 23:00:19.250661 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.250500    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca\") pod \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\" (UID: \"33f18d49-b7c0-4554-9a59-b16bb9cfa220\") "
Oct 30 23:00:19.250946 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.250926    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca" pod "33f18d49-b7c0-4554-9a59-b16bb9cfa220" (UID: "33f18d49-b7c0-4554-9a59-b16bb9cfa220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:19.252568 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.252545    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc" pod "33f18d49-b7c0-4554-9a59-b16bb9cfa220" (UID: "33f18d49-b7c0-4554-9a59-b16bb9cfa220"). InnerVolumeSpecName "kube-api-access-724dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:19.352405 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.352373    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-724dc\" (UniqueName: \"kubernetes.io/projected/33f18d49-b7c0-4554-9a59-b16bb9cfa220-kube-api-access-724dc\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:00:19.352405 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.352403    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f18d49-b7c0-4554-9a59-b16bb9cfa220-maas-api-service-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:00:19.784651 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.784613    5028 scope.go:122] "RemoveContainer" containerID="458b3084202df6ce218cc0647ff92fd6a2849ae6b305383761f279732c6ce89a"
Oct 30 23:00:19.794134 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:19.794112    5028 scope.go:122] "RemoveContainer" containerID="f9706dee5bfa49ae119cbcd58d6db338f17c061d17bb75a64d47252626c72aba"
Oct 30 23:00:20.011121 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.011092    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx" event={"ID":"33f18d49-b7c0-4554-9a59-b16bb9cfa220","Type":"ContainerDied","Data":"b9e8936aec46f9455d3346e500aaedadc42c2ba2929f1f161893c108ee3f30c3"}
Oct 30 23:00:20.011318 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.011143    5028 scope.go:122] "RemoveContainer" containerID="c09ef1e2360b688b50c1a0c39989306779165e325f9daf323d5f34d181745147"
Oct 30 23:00:20.011318 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.011101    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"
Oct 30 23:00:20.120928 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.120852    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:20.169781 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.169748    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"]
Oct 30 23:00:20.175118 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.175089    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-phnzx"]
Oct 30 23:00:20.287199 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:20.287160    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:00:21.289942 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:00:21.289898    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" path="/var/lib/kubelet/pods/33f18d49-b7c0-4554-9a59-b16bb9cfa220/volumes"
Oct 30 23:00:28.435287 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:00:28.435243    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:00:58.435468 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:00:58.435429    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:01:10.606568 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:10.606540    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 23:01:10.646389 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:10.646359    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 23:01:18.962193 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962159    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"]
Oct 30 23:01:18.962635 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962620    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" containerName="manager"
Oct 30 23:01:18.962681 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962635    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" containerName="manager"
Oct 30 23:01:18.962681 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962665    5028 cpu_manager.go:414] "containerMap: removing container" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962681 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962670    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962681 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962678    5028 cpu_manager.go:414] "containerMap: removing container" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962707    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962718    5028 cpu_manager.go:414] "containerMap: removing container" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962723    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962793    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.962850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962806    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6194d8-8f7b-46aa-a3dc-c98949a2fba9" containerName="manager"
Oct 30 23:01:18.962850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.962818    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:18.965974 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.965956    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:18.997850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.997812    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb45a923-6e92-4af4-b616-7309a32c021e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6rk9z\" (UID: \"cb45a923-6e92-4af4-b616-7309a32c021e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:18.997981 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:18.997937    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4cts\" (UniqueName: \"kubernetes.io/projected/cb45a923-6e92-4af4-b616-7309a32c021e-kube-api-access-q4cts\") pod \"kuadrant-operator-controller-manager-678d87766f-6rk9z\" (UID: \"cb45a923-6e92-4af4-b616-7309a32c021e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:19.017927 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.017900    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"]
Oct 30 23:01:19.062094 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.062060    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nhrgr\""
Oct 30 23:01:19.098748 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.098715    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb45a923-6e92-4af4-b616-7309a32c021e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6rk9z\" (UID: \"cb45a923-6e92-4af4-b616-7309a32c021e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:19.098904 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.098805    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cts\" (UniqueName: \"kubernetes.io/projected/cb45a923-6e92-4af4-b616-7309a32c021e-kube-api-access-q4cts\") pod \"kuadrant-operator-controller-manager-678d87766f-6rk9z\" (UID: \"cb45a923-6e92-4af4-b616-7309a32c021e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:19.099094 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.099075    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb45a923-6e92-4af4-b616-7309a32c021e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6rk9z\" (UID: \"cb45a923-6e92-4af4-b616-7309a32c021e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:19.158450 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.158415    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cts\" (UniqueName: \"kubernetes.io/projected/cb45a923-6e92-4af4-b616-7309a32c021e-kube-api-access-q4cts\") pod \"kuadrant-operator-controller-manager-678d87766f-6rk9z\" (UID: \"cb45a923-6e92-4af4-b616-7309a32c021e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:19.279913 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.279825    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:19.751158 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.751081    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"]
Oct 30 23:01:19.752142 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:01:19.752115    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb45a923_6e92_4af4_b616_7309a32c021e.slice/crio-29fdd06b92fa02928ccb03ff22a59fa0e51e40be4f6357f566c4e344ee864baa WatchSource:0}: Error finding container 29fdd06b92fa02928ccb03ff22a59fa0e51e40be4f6357f566c4e344ee864baa: Status 404 returned error can't find the container with id 29fdd06b92fa02928ccb03ff22a59fa0e51e40be4f6357f566c4e344ee864baa
Oct 30 23:01:19.832733 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.832705    5028 scope.go:122] "RemoveContainer" containerID="6e3aa96ccc9dcfd739ff98edd55fb675f817d32873bad7c335f49784620a1eb0"
Oct 30 23:01:19.842058 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:19.842033    5028 scope.go:122] "RemoveContainer" containerID="0e00e3d79fb050c974d51a9b313d5612703cb402e0935ced5586f43d8235c5f6"
Oct 30 23:01:20.379306 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:20.379263    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z" event={"ID":"cb45a923-6e92-4af4-b616-7309a32c021e","Type":"ContainerStarted","Data":"045beb631f727b5ecb505deb400de6b59aeb50726c555d2462c540047e7f561e"}
Oct 30 23:01:20.379725 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:20.379323    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z" event={"ID":"cb45a923-6e92-4af4-b616-7309a32c021e","Type":"ContainerStarted","Data":"29fdd06b92fa02928ccb03ff22a59fa0e51e40be4f6357f566c4e344ee864baa"}
Oct 30 23:01:20.379725 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:20.379352    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:20.476720 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:20.476654    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:01:20.496032 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:20.495983    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z" podStartSLOduration=2.495966489 podStartE2EDuration="2.495966489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:20.494016516 +0000 UTC m=+3611.343529488" watchObservedRunningTime="2026-10-30 23:01:20.495966489 +0000 UTC m=+3611.345479462"
Oct 30 23:01:28.435709 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:01:28.435652    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:01:31.388236 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:31.388205    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6rk9z"
Oct 30 23:01:44.731639 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.731594    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"]
Oct 30 23:01:44.732756 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.732421    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f18d49-b7c0-4554-9a59-b16bb9cfa220" containerName="cleanup"
Oct 30 23:01:44.736253 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.736232    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.789043 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.789011    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"]
Oct 30 23:01:44.808041 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.808013    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.808219 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.808077    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.808219 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.808182    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cqk8\" (UniqueName: \"kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.832631 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.832597    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:01:44.832845 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.832733    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:01:44.832845 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.832788    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:01:44.833902 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.833889    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:01:44.909319 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.909288    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.909492 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.909334    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cqk8\" (UniqueName: \"kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.909492 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.909412    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.910133 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.910107    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.911750 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.911727    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:44.968819 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:44.968789    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cqk8\" (UniqueName: \"kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8\") pod \"maas-api-revoke-sub-d576e68b-zl4wm\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:45.053709 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:45.053599    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:45.425465 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:45.425438    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"]
Oct 30 23:01:45.426295 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:01:45.426272    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2de7149_0c84_4c4a_bb39_37dd044f14af.slice/crio-207f739fd4aeacba6c8cba348648ad2596e5c6027ef24c8daf35f062975ff0fa WatchSource:0}: Error finding container 207f739fd4aeacba6c8cba348648ad2596e5c6027ef24c8daf35f062975ff0fa: Status 404 returned error can't find the container with id 207f739fd4aeacba6c8cba348648ad2596e5c6027ef24c8daf35f062975ff0fa
Oct 30 23:01:45.550175 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:45.550141    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm" event={"ID":"e2de7149-0c84-4c4a-bb39-37dd044f14af","Type":"ContainerStarted","Data":"207f739fd4aeacba6c8cba348648ad2596e5c6027ef24c8daf35f062975ff0fa"}
Oct 30 23:01:46.557961 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:46.557878    5028 generic.go:356] "Generic (PLEG): container finished" podID="e2de7149-0c84-4c4a-bb39-37dd044f14af" containerID="c407ec172e2e7b5a0da7d67b2254e805005c23b4e84d3a0d3e707ecde04e1534" exitCode=0
Oct 30 23:01:46.558334 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:46.557959    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm" event={"ID":"e2de7149-0c84-4c4a-bb39-37dd044f14af","Type":"ContainerDied","Data":"c407ec172e2e7b5a0da7d67b2254e805005c23b4e84d3a0d3e707ecde04e1534"}
Oct 30 23:01:47.704416 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.704378    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:47.839341 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.839257    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca\") pod \"e2de7149-0c84-4c4a-bb39-37dd044f14af\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") "
Oct 30 23:01:47.839518 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.839395    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca\") pod \"e2de7149-0c84-4c4a-bb39-37dd044f14af\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") "
Oct 30 23:01:47.839518 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.839444    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8\" (UniqueName: \"kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8\") pod \"e2de7149-0c84-4c4a-bb39-37dd044f14af\" (UID: \"e2de7149-0c84-4c4a-bb39-37dd044f14af\") "
Oct 30 23:01:47.839630 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.839606    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca" pod "e2de7149-0c84-4c4a-bb39-37dd044f14af" (UID: "e2de7149-0c84-4c4a-bb39-37dd044f14af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:47.839765 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.839751    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:47.841640 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.841610    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8" pod "e2de7149-0c84-4c4a-bb39-37dd044f14af" (UID: "e2de7149-0c84-4c4a-bb39-37dd044f14af"). InnerVolumeSpecName "kube-api-access-4cqk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:47.841640 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.841622    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca" pod "e2de7149-0c84-4c4a-bb39-37dd044f14af" (UID: "e2de7149-0c84-4c4a-bb39-37dd044f14af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:47.940746 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.940714    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2de7149-0c84-4c4a-bb39-37dd044f14af-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:47.940746 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:47.940742    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cqk8\" (UniqueName: \"kubernetes.io/projected/e2de7149-0c84-4c4a-bb39-37dd044f14af-kube-api-access-4cqk8\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:48.570644 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:48.570604    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm" event={"ID":"e2de7149-0c84-4c4a-bb39-37dd044f14af","Type":"ContainerDied","Data":"207f739fd4aeacba6c8cba348648ad2596e5c6027ef24c8daf35f062975ff0fa"}
Oct 30 23:01:48.570644 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:48.570622    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"
Oct 30 23:01:48.570644 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:48.570653    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207f739fd4aeacba6c8cba348648ad2596e5c6027ef24c8daf35f062975ff0fa"
Oct 30 23:01:49.049015 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:49.048970    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"]
Oct 30 23:01:49.051788 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:49.051765    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f44xx"]
Oct 30 23:01:49.289747 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:01:49.289668    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a097bc2-bd8a-413f-ab1e-4053630dbd8e" path="/var/lib/kubelet/pods/7a097bc2-bd8a-413f-ab1e-4053630dbd8e/volumes"
Oct 30 23:01:58.435151 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:01:58.435108    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:02:00.066256 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:00.066217    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"]
Oct 30 23:02:00.111407 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:00.111375    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t9htp"]
Oct 30 23:02:01.289749 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:01.289680    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6" path="/var/lib/kubelet/pods/cbc1a7c8-e5c8-4569-9c1d-235457f6c1c6/volumes"
Oct 30 23:02:13.810138 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.810101    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"]
Oct 30 23:02:13.810598 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.810584    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e2de7149-0c84-4c4a-bb39-37dd044f14af" containerName="revoke-keys"
Oct 30 23:02:13.810646 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.810599    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2de7149-0c84-4c4a-bb39-37dd044f14af" containerName="revoke-keys"
Oct 30 23:02:13.810731 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.810720    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2de7149-0c84-4c4a-bb39-37dd044f14af" containerName="revoke-keys"
Oct 30 23:02:13.813957 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.813941    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.842575 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.842546    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.842764 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.842661    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bcn\" (UniqueName: \"kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.842764 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.842755    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.869406 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.869378    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"]
Oct 30 23:02:13.912995 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.912970    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:02:13.913276 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.913252    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:02:13.913373 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.913255    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:02:13.913373 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.913286    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:02:13.944010 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.943983    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.944162 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.944045    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.944162 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.944103    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bcn\" (UniqueName: \"kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.944846 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.944826    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:13.946469 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:13.946446    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:14.003355 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:14.003319    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bcn\" (UniqueName: \"kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn\") pod \"maas-api-revoke-sub-d2bc93bc-559hw\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:14.128782 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:14.128694    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:14.496146 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:14.496109    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"]
Oct 30 23:02:14.496441 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:02:14.496416    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2547f9_8add_4d73_b248_b1068f480d0b.slice/crio-9e527a06ed3652803b90342f9cad55b2ca9aaa89075c0ec6a32f5c39ebfc1433 WatchSource:0}: Error finding container 9e527a06ed3652803b90342f9cad55b2ca9aaa89075c0ec6a32f5c39ebfc1433: Status 404 returned error can't find the container with id 9e527a06ed3652803b90342f9cad55b2ca9aaa89075c0ec6a32f5c39ebfc1433
Oct 30 23:02:14.752427 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:14.752360    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw" event={"ID":"5c2547f9-8add-4d73-b248-b1068f480d0b","Type":"ContainerStarted","Data":"9e527a06ed3652803b90342f9cad55b2ca9aaa89075c0ec6a32f5c39ebfc1433"}
Oct 30 23:02:15.759741 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:15.759682    5028 generic.go:356] "Generic (PLEG): container finished" podID="5c2547f9-8add-4d73-b248-b1068f480d0b" containerID="fc959924b6b08965d1f1024686e061a2f24605a97e6d0e6f49b358e98563051c" exitCode=0
Oct 30 23:02:15.760140 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:15.759780    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw" event={"ID":"5c2547f9-8add-4d73-b248-b1068f480d0b","Type":"ContainerDied","Data":"fc959924b6b08965d1f1024686e061a2f24605a97e6d0e6f49b358e98563051c"}
Oct 30 23:02:16.911527 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.911505    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:16.972804 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.972769    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca\") pod \"5c2547f9-8add-4d73-b248-b1068f480d0b\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") "
Oct 30 23:02:16.972995 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.972825    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn\" (UniqueName: \"kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn\") pod \"5c2547f9-8add-4d73-b248-b1068f480d0b\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") "
Oct 30 23:02:16.972995 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.972904    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca\") pod \"5c2547f9-8add-4d73-b248-b1068f480d0b\" (UID: \"5c2547f9-8add-4d73-b248-b1068f480d0b\") "
Oct 30 23:02:16.973140 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.973115    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca" pod "5c2547f9-8add-4d73-b248-b1068f480d0b" (UID: "5c2547f9-8add-4d73-b248-b1068f480d0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:16.973257 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.973231    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:02:16.974973 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.974948    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca" pod "5c2547f9-8add-4d73-b248-b1068f480d0b" (UID: "5c2547f9-8add-4d73-b248-b1068f480d0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:16.975091 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:16.975060    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn" pod "5c2547f9-8add-4d73-b248-b1068f480d0b" (UID: "5c2547f9-8add-4d73-b248-b1068f480d0b"). InnerVolumeSpecName "kube-api-access-p5bcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:17.074662 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:17.074580    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5bcn\" (UniqueName: \"kubernetes.io/projected/5c2547f9-8add-4d73-b248-b1068f480d0b-kube-api-access-p5bcn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:02:17.074662 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:17.074610    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2547f9-8add-4d73-b248-b1068f480d0b-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:02:17.773476 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:17.773446    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"
Oct 30 23:02:17.773664 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:17.773481    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw" event={"ID":"5c2547f9-8add-4d73-b248-b1068f480d0b","Type":"ContainerDied","Data":"9e527a06ed3652803b90342f9cad55b2ca9aaa89075c0ec6a32f5c39ebfc1433"}
Oct 30 23:02:17.773664 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:17.773532    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e527a06ed3652803b90342f9cad55b2ca9aaa89075c0ec6a32f5c39ebfc1433"
Oct 30 23:02:19.881046 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:19.881008    5028 scope.go:122] "RemoveContainer" containerID="a4417f1cfde97a90588595434d482c3a4c6097ec4048cb0d1e2e0cbf25976aad"
Oct 30 23:02:19.893251 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:19.893225    5028 scope.go:122] "RemoveContainer" containerID="506e3c8e2e0772eba41ef0d4e8bb2c848022a6bd88d2a1532030d7f88c16f2cf"
Oct 30 23:02:20.848330 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:20.848289    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:26.124875 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:26.124843    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:26.848326 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:26.848286    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:27.085218 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:27.085182    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:28.435048 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:02:28.435010    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:02:30.736742 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:30.736704    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:30.926078 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:30.926045    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:31.093068 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:31.092974    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:33.918877 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:33.918842    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:34.074547 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.074514    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:34.299772 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.299734    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"]
Oct 30 23:02:34.300194 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.300180    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5c2547f9-8add-4d73-b248-b1068f480d0b" containerName="revoke-keys"
Oct 30 23:02:34.300249 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.300195    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2547f9-8add-4d73-b248-b1068f480d0b" containerName="revoke-keys"
Oct 30 23:02:34.300290 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.300257    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2547f9-8add-4d73-b248-b1068f480d0b" containerName="revoke-keys"
Oct 30 23:02:34.305069 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.305049    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.357103 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.357071    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2e092e\""
Oct 30 23:02:34.362820 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.362791    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"]
Oct 30 23:02:34.404728 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.404656    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b38a71aa-dockercfg-v9wc2\""
Oct 30 23:02:34.452559 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.452519    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclh4\" (UniqueName: \"kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4\") pod \"payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.452757 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.452592    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.553848 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.553762    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclh4\" (UniqueName: \"kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4\") pod \"payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.553848 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.553819    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.554558 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.554536    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.613390 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.613357    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclh4\" (UniqueName: \"kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4\") pod \"payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.618181 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.618158    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:34.989548 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:34.989518    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"]
Oct 30 23:02:34.990806 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:02:34.990777    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab1df83_9340_4dd8_a3c0_a0ce660c2298.slice/crio-8b848489a4e99c0b186ca24d1b257ace3b16debe2b8a8b0ec2224766c4df879e WatchSource:0}: Error finding container 8b848489a4e99c0b186ca24d1b257ace3b16debe2b8a8b0ec2224766c4df879e: Status 404 returned error can't find the container with id 8b848489a4e99c0b186ca24d1b257ace3b16debe2b8a8b0ec2224766c4df879e
Oct 30 23:02:35.880628 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:35.880589    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" event={"ID":"7ab1df83-9340-4dd8-a3c0-a0ce660c2298","Type":"ContainerStarted","Data":"b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f"}
Oct 30 23:02:35.880850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:35.880656    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:35.880850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:35.880673    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" event={"ID":"7ab1df83-9340-4dd8-a3c0-a0ce660c2298","Type":"ContainerStarted","Data":"8b848489a4e99c0b186ca24d1b257ace3b16debe2b8a8b0ec2224766c4df879e"}
Oct 30 23:02:35.993534 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:35.993479    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" podStartSLOduration=1.993465138 podStartE2EDuration="1.993465138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:35.991839481 +0000 UTC m=+3686.841352455" watchObservedRunningTime="2026-10-30 23:02:35.993465138 +0000 UTC m=+3686.842978110"
Oct 30 23:02:36.698484 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:36.698452    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:36.857853 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:36.857820    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:37.029671 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:37.029634    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:41.371172 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:41.371134    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:46.069126 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:46.069083    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:46.890217 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:46.890183    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:02:50.644421 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:50.644377    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:52.129573 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:52.129528    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:55.452843 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:55.452804    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:55.612535 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:55.612497    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:56.210441 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:56.210406    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:56.404894 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:56.404862    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:56.563369 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:02:56.563290    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:02:58.437198 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:02:58.437159    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:03:00.121322 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:00.121282    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:00.277353 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:00.277319    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:04.644260 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:04.644220    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:09.141847 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:09.141813    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:13.736920 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:13.736884    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:18.247245 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:18.247210    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:19.923889 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:19.923814    5028 scope.go:122] "RemoveContainer" containerID="11e31960a903522244084c2877fb238af00d0a06330d6a9942b590d80fc3c71a"
Oct 30 23:03:19.933331 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:19.933308    5028 scope.go:122] "RemoveContainer" containerID="8762e12a39400216bb3c086d933a7e2cd06d6c8810972697369b680b7d680df1"
Oct 30 23:03:19.941785 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:19.941759    5028 scope.go:122] "RemoveContainer" containerID="91d4fdb34eb729b330bd3e3c3bcf5bc9e4369e683a897cf79084961cc46cedca"
Oct 30 23:03:21.147949 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:21.147918    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:21.301597 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:21.301561    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:22.948844 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:22.948813    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:27.441269 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:27.441231    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:28.436070 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:03:28.436023    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:03:32.089225 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:32.089183    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:36.561057 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:36.561008    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:41.144440 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:41.144400    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:45.656165 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:45.656127    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:46.554282 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.554243    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"]
Oct 30 23:03:46.557977 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.557955    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.613082 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.613048    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"]
Oct 30 23:03:46.652227 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.652185    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:46.656658 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.656636    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:46.657196 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.657178    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:46.657258 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.657245    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:03:46.686393 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.686359    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.686593 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.686429    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.686659 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.686594    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv89\" (UniqueName: \"kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.787992 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.787949    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv89\" (UniqueName: \"kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.788233 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.788031    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.788233 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.788131    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.788843 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.788821    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.790448 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.790422    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.847653 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.847578    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv89\" (UniqueName: \"kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89\") pod \"maas-api-revoke-sub-89007d93-7cd64\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:46.871113 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:46.871082    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:47.241299 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:47.241264    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"]
Oct 30 23:03:47.241938 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:03:47.241912    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae3bdd5_b488_410f_b539_064d3e4d212f.slice/crio-0985d2c8392cea37bdfe82899a2b81ffea50fcec42b55671b6c02241a8e02354 WatchSource:0}: Error finding container 0985d2c8392cea37bdfe82899a2b81ffea50fcec42b55671b6c02241a8e02354: Status 404 returned error can't find the container with id 0985d2c8392cea37bdfe82899a2b81ffea50fcec42b55671b6c02241a8e02354
Oct 30 23:03:47.324149 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:47.324113    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64" event={"ID":"aae3bdd5-b488-410f-b539-064d3e4d212f","Type":"ContainerStarted","Data":"0985d2c8392cea37bdfe82899a2b81ffea50fcec42b55671b6c02241a8e02354"}
Oct 30 23:03:48.332932 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:48.332827    5028 generic.go:356] "Generic (PLEG): container finished" podID="aae3bdd5-b488-410f-b539-064d3e4d212f" containerID="39b36f89078c93c712559f6f458b943607c03495d33500fded40929d9845cdc5" exitCode=0
Oct 30 23:03:48.333380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:48.332944    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64" event={"ID":"aae3bdd5-b488-410f-b539-064d3e4d212f","Type":"ContainerDied","Data":"39b36f89078c93c712559f6f458b943607c03495d33500fded40929d9845cdc5"}
Oct 30 23:03:49.604826 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.604751    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:49.720013 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.719979    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89\" (UniqueName: \"kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89\") pod \"aae3bdd5-b488-410f-b539-064d3e4d212f\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") "
Oct 30 23:03:49.720317 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.720288    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca\") pod \"aae3bdd5-b488-410f-b539-064d3e4d212f\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") "
Oct 30 23:03:49.720461 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.720343    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca\") pod \"aae3bdd5-b488-410f-b539-064d3e4d212f\" (UID: \"aae3bdd5-b488-410f-b539-064d3e4d212f\") "
Oct 30 23:03:49.721160 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.721135    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca" pod "aae3bdd5-b488-410f-b539-064d3e4d212f" (UID: "aae3bdd5-b488-410f-b539-064d3e4d212f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:49.726038 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.724807    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca" pod "aae3bdd5-b488-410f-b539-064d3e4d212f" (UID: "aae3bdd5-b488-410f-b539-064d3e4d212f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:49.726038 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.724925    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89" pod "aae3bdd5-b488-410f-b539-064d3e4d212f" (UID: "aae3bdd5-b488-410f-b539-064d3e4d212f"). InnerVolumeSpecName "kube-api-access-6rv89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:49.822094 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.822048    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv89\" (UniqueName: \"kubernetes.io/projected/aae3bdd5-b488-410f-b539-064d3e4d212f-kube-api-access-6rv89\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:03:49.822094 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.822084    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:03:49.822094 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:49.822095    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae3bdd5-b488-410f-b539-064d3e4d212f-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:03:50.228255 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:50.228197    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:50.347431 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:50.347398    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"
Oct 30 23:03:50.347604 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:50.347427    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64" event={"ID":"aae3bdd5-b488-410f-b539-064d3e4d212f","Type":"ContainerDied","Data":"0985d2c8392cea37bdfe82899a2b81ffea50fcec42b55671b6c02241a8e02354"}
Oct 30 23:03:50.347604 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:50.347485    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0985d2c8392cea37bdfe82899a2b81ffea50fcec42b55671b6c02241a8e02354"
Oct 30 23:03:54.737847 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:54.737796    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:57.686150 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:57.686096    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:58.129622 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.129583    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"]
Oct 30 23:03:58.130050 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.130036    5028 cpu_manager.go:414] "containerMap: removing container" podUID="aae3bdd5-b488-410f-b539-064d3e4d212f" containerName="revoke-keys"
Oct 30 23:03:58.130100 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.130051    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae3bdd5-b488-410f-b539-064d3e4d212f" containerName="revoke-keys"
Oct 30 23:03:58.130152 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.130143    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae3bdd5-b488-410f-b539-064d3e4d212f" containerName="revoke-keys"
Oct 30 23:03:58.133645 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.133629    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.184881 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.184854    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:03:58.185068 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.184865    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:03:58.188231 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.188205    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"]
Oct 30 23:03:58.231780 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.231745    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:03:58.231780 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.231755    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:03:58.305024 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.304990    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.305197 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.305048    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzr2j\" (UniqueName: \"kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.305238 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.305188    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.378981 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.378949    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:03:58.406225 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.406135    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.406426 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.406222    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.406426 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.406277    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzr2j\" (UniqueName: \"kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.406895 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.406869    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.408779 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.408752    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.434438 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:03:58.434399    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:03:58.465271 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.465237    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzr2j\" (UniqueName: \"kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:58.747002 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:58.746969    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:03:59.118729 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:59.118675    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"]
Oct 30 23:03:59.120429 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:03:59.120385    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60a2b52_38f3_4ddf_8af3_56c3a196ab90.slice/crio-81c5f2a1c9bccf80543ef032d23c4966da613b47429cec8200e531da7389da65 WatchSource:0}: Error finding container 81c5f2a1c9bccf80543ef032d23c4966da613b47429cec8200e531da7389da65: Status 404 returned error can't find the container with id 81c5f2a1c9bccf80543ef032d23c4966da613b47429cec8200e531da7389da65
Oct 30 23:03:59.402054 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:59.401967    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h" event={"ID":"a60a2b52-38f3-4ddf-8af3-56c3a196ab90","Type":"ContainerStarted","Data":"81c5f2a1c9bccf80543ef032d23c4966da613b47429cec8200e531da7389da65"}
Oct 30 23:03:59.436738 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:03:59.436709    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:00.410608 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:00.410501    5028 generic.go:356] "Generic (PLEG): container finished" podID="a60a2b52-38f3-4ddf-8af3-56c3a196ab90" containerID="0726535e3695850ea427cc292b52ec2bc67dd9c7409d5aab044dec54219e44e7" exitCode=0
Oct 30 23:04:00.411113 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:00.410744    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h" event={"ID":"a60a2b52-38f3-4ddf-8af3-56c3a196ab90","Type":"ContainerDied","Data":"0726535e3695850ea427cc292b52ec2bc67dd9c7409d5aab044dec54219e44e7"}
Oct 30 23:04:01.575129 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.575100    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:04:01.741972 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.741930    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j\" (UniqueName: \"kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j\") pod \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") "
Oct 30 23:04:01.741972 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.741976    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca\") pod \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") "
Oct 30 23:04:01.742263 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.742088    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca\") pod \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\" (UID: \"a60a2b52-38f3-4ddf-8af3-56c3a196ab90\") "
Oct 30 23:04:01.742469 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.742443    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca" pod "a60a2b52-38f3-4ddf-8af3-56c3a196ab90" (UID: "a60a2b52-38f3-4ddf-8af3-56c3a196ab90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:01.744275 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.744240    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca" pod "a60a2b52-38f3-4ddf-8af3-56c3a196ab90" (UID: "a60a2b52-38f3-4ddf-8af3-56c3a196ab90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:01.744380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.744308    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j" pod "a60a2b52-38f3-4ddf-8af3-56c3a196ab90" (UID: "a60a2b52-38f3-4ddf-8af3-56c3a196ab90"). InnerVolumeSpecName "kube-api-access-vzr2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:01.843588 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.843551    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzr2j\" (UniqueName: \"kubernetes.io/projected/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-kube-api-access-vzr2j\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:01.843588 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.843585    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:01.843588 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:01.843597    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60a2b52-38f3-4ddf-8af3-56c3a196ab90-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:02.423874 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:02.423839    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"
Oct 30 23:04:02.424099 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:02.423879    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h" event={"ID":"a60a2b52-38f3-4ddf-8af3-56c3a196ab90","Type":"ContainerDied","Data":"81c5f2a1c9bccf80543ef032d23c4966da613b47429cec8200e531da7389da65"}
Oct 30 23:04:02.424099 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:02.423929    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c5f2a1c9bccf80543ef032d23c4966da613b47429cec8200e531da7389da65"
Oct 30 23:04:02.722867 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:02.722829    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:04.330548 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:04.330515    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:07.046794 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:07.046759    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"]
Oct 30 23:04:07.093652 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:07.093615    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3999c1b9-bng87"]
Oct 30 23:04:07.289520 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:07.289489    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45ad654-117f-463d-a2e0-04935931ea9e" path="/var/lib/kubelet/pods/e45ad654-117f-463d-a2e0-04935931ea9e/volumes"
Oct 30 23:04:08.846586 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:08.846549    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:09.684520 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:09.684484    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"]
Oct 30 23:04:09.950872 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:09.950785    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:10.127870 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:10.127840    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:13.365946 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:13.365907    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:14.685452 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:14.685391    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerName="payload-pre-processing" containerID="cri-o://b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f" gracePeriod=25
Oct 30 23:04:15.505105 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:15.505063    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:15.658609 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:15.658574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:16.888308 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:16.888271    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:17.304586 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:17.304541    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:18.723743 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:18.723709    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:18.992933 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:18.992849    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:19.293415 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:19.293325    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"]
Oct 30 23:04:19.340639 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:19.340609    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2e092e-djf8h"]
Oct 30 23:04:19.745832 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:19.745799    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:21.289755 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:21.289710    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60a2b52-38f3-4ddf-8af3-56c3a196ab90" path="/var/lib/kubelet/pods/a60a2b52-38f3-4ddf-8af3-56c3a196ab90/volumes"
Oct 30 23:04:21.497412 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:21.497377    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.745893 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:25.745853    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:25.907901 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:25.907865    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:26.137670 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:26.137541    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:26.888429 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:26.888389    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:04:28.437447 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:04:28.437411    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:04:33.508554 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:33.508519    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:34.961047 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:34.961014    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:04:35.095385 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.095347    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume\") pod \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") "
Oct 30 23:04:35.095560 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.095461    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4\" (UniqueName: \"kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4\") pod \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\" (UID: \"7ab1df83-9340-4dd8-a3c0-a0ce660c2298\") "
Oct 30 23:04:35.095767 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.095745    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume" pod "7ab1df83-9340-4dd8-a3c0-a0ce660c2298" (UID: "7ab1df83-9340-4dd8-a3c0-a0ce660c2298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:35.097738 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.097709    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4" pod "7ab1df83-9340-4dd8-a3c0-a0ce660c2298" (UID: "7ab1df83-9340-4dd8-a3c0-a0ce660c2298"). InnerVolumeSpecName "kube-api-access-lclh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:35.197234 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.197197    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:35.197234 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.197229    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lclh4\" (UniqueName: \"kubernetes.io/projected/7ab1df83-9340-4dd8-a3c0-a0ce660c2298-kube-api-access-lclh4\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:35.622979 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.622896    5028 generic.go:356] "Generic (PLEG): container finished" podID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerID="b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f" exitCode=0
Oct 30 23:04:35.622979 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.622948    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"
Oct 30 23:04:35.622979 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.622955    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" event={"ID":"7ab1df83-9340-4dd8-a3c0-a0ce660c2298","Type":"ContainerDied","Data":"b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f"}
Oct 30 23:04:35.623231 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.623006    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl" event={"ID":"7ab1df83-9340-4dd8-a3c0-a0ce660c2298","Type":"ContainerDied","Data":"8b848489a4e99c0b186ca24d1b257ace3b16debe2b8a8b0ec2224766c4df879e"}
Oct 30 23:04:35.623231 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.623031    5028 scope.go:122] "RemoveContainer" containerID="b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f"
Oct 30 23:04:35.633316 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.633290    5028 scope.go:122] "RemoveContainer" containerID="b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f"
Oct 30 23:04:35.633641 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:04:35.633619    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f\": container with ID starting with b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f not found: ID does not exist" containerID="b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f"
Oct 30 23:04:35.633737 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.633651    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f"} err="failed to get container status \"b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f\": rpc error: code = NotFound desc = could not find container \"b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f\": container with ID starting with b0892a40088d7b7f985399ed22521cdf7fe515ababb1447a5b48da4fc28d051f not found: ID does not exist"
Oct 30 23:04:35.781323 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.781280    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"]
Oct 30 23:04:35.785450 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:35.785417    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2e092e-78f674nhnl"]
Oct 30 23:04:37.290223 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:37.290191    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" path="/var/lib/kubelet/pods/7ab1df83-9340-4dd8-a3c0-a0ce660c2298/volumes"
Oct 30 23:04:38.854869 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:38.854830    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:40.095454 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:40.095416    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:04:44.048249 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:44.048209    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"]
Oct 30 23:04:44.095155 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:44.095110    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8g"]
Oct 30 23:04:45.291992 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:04:45.291956    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d6a054-2fc5-4ac3-9d24-5a255831b93d" path="/var/lib/kubelet/pods/18d6a054-2fc5-4ac3-9d24-5a255831b93d/volumes"
Oct 30 23:04:58.434801 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:04:58.434719    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:05:03.656221 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:03.656185    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:17.657836 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:17.657800    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:19.981394 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:19.981356    5028 scope.go:122] "RemoveContainer" containerID="e8e7f3a5a46579f0b085b97bef796015937b56bec95d6374bea8363e11c2598d"
Oct 30 23:05:19.990914 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:19.990894    5028 scope.go:122] "RemoveContainer" containerID="81eb722f4469144a2bf2dde13965e07879720c2440823e6c4ba7c6e93982988b"
Oct 30 23:05:21.857971 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:21.857904    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:28.436028 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:05:28.435991    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:05:39.641430 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.641391    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"]
Oct 30 23:05:39.642055 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.642032    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerName="payload-pre-processing"
Oct 30 23:05:39.642155 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.642057    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerName="payload-pre-processing"
Oct 30 23:05:39.642155 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.642102    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a60a2b52-38f3-4ddf-8af3-56c3a196ab90" containerName="revoke-keys"
Oct 30 23:05:39.642155 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.642111    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60a2b52-38f3-4ddf-8af3-56c3a196ab90" containerName="revoke-keys"
Oct 30 23:05:39.642319 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.642243    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab1df83-9340-4dd8-a3c0-a0ce660c2298" containerName="payload-pre-processing"
Oct 30 23:05:39.642319 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.642263    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60a2b52-38f3-4ddf-8af3-56c3a196ab90" containerName="revoke-keys"
Oct 30 23:05:39.646930 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.646908    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.701351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.701322    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"]
Oct 30 23:05:39.744306 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.744275    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:05:39.744306 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.744312    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:05:39.744535 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.744312    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:05:39.744535 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.744409    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:05:39.810190 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.810152    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.810375 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.810219    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.810375 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.810315    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphcb\" (UniqueName: \"kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.911583 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.911483    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.911767 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.911679    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.911767 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.911753    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphcb\" (UniqueName: \"kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.912407 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.912380    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.914163 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.914140    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:39.970804 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:39.970769    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphcb\" (UniqueName: \"kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb\") pod \"maas-api-revoke-sub-452b6d7b-dcdjt\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:40.260778 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:40.260723    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:40.628638 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:40.628608    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"]
Oct 30 23:05:40.629934 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:05:40.629906    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5881263_63ac_4c33_866f_1f4add3504c6.slice/crio-7f458f235d5a0a201de58bf840384b25b570f8d11f50e51bc4b3fa6a11531642 WatchSource:0}: Error finding container 7f458f235d5a0a201de58bf840384b25b570f8d11f50e51bc4b3fa6a11531642: Status 404 returned error can't find the container with id 7f458f235d5a0a201de58bf840384b25b570f8d11f50e51bc4b3fa6a11531642
Oct 30 23:05:41.092948 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:41.092912    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt" event={"ID":"b5881263-63ac-4c33-866f-1f4add3504c6","Type":"ContainerStarted","Data":"b45ca9dc3cd0fe85f7376c7df6108c115b02b9a4f1de448a8fd24b0ba701b1d9"}
Oct 30 23:05:41.093348 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:41.092969    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt" event={"ID":"b5881263-63ac-4c33-866f-1f4add3504c6","Type":"ContainerStarted","Data":"7f458f235d5a0a201de58bf840384b25b570f8d11f50e51bc4b3fa6a11531642"}
Oct 30 23:05:41.204141 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:41.203998    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt" podStartSLOduration=2.203980409 podStartE2EDuration="2.203980409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:41.203818356 +0000 UTC m=+3872.053331332" watchObservedRunningTime="2026-10-30 23:05:41.203980409 +0000 UTC m=+3872.053493383"
Oct 30 23:05:42.104668 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:42.104617    5028 generic.go:356] "Generic (PLEG): container finished" podID="b5881263-63ac-4c33-866f-1f4add3504c6" containerID="b45ca9dc3cd0fe85f7376c7df6108c115b02b9a4f1de448a8fd24b0ba701b1d9" exitCode=0
Oct 30 23:05:42.105241 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:42.104812    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt" event={"ID":"b5881263-63ac-4c33-866f-1f4add3504c6","Type":"ContainerDied","Data":"b45ca9dc3cd0fe85f7376c7df6108c115b02b9a4f1de448a8fd24b0ba701b1d9"}
Oct 30 23:05:44.248656 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.248295    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:44.303785 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.303137    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt" event={"ID":"b5881263-63ac-4c33-866f-1f4add3504c6","Type":"ContainerDied","Data":"7f458f235d5a0a201de58bf840384b25b570f8d11f50e51bc4b3fa6a11531642"}
Oct 30 23:05:44.303785 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.303195    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f458f235d5a0a201de58bf840384b25b570f8d11f50e51bc4b3fa6a11531642"
Oct 30 23:05:44.303785 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.303267    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"
Oct 30 23:05:44.303785 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.303634    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb\" (UniqueName: \"kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb\") pod \"b5881263-63ac-4c33-866f-1f4add3504c6\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") "
Oct 30 23:05:44.303785 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.303723    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca\") pod \"b5881263-63ac-4c33-866f-1f4add3504c6\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") "
Oct 30 23:05:44.304262 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.303820    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca\") pod \"b5881263-63ac-4c33-866f-1f4add3504c6\" (UID: \"b5881263-63ac-4c33-866f-1f4add3504c6\") "
Oct 30 23:05:44.308725 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.308215    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca" pod "b5881263-63ac-4c33-866f-1f4add3504c6" (UID: "b5881263-63ac-4c33-866f-1f4add3504c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:44.341410 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.341369    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb" pod "b5881263-63ac-4c33-866f-1f4add3504c6" (UID: "b5881263-63ac-4c33-866f-1f4add3504c6"). InnerVolumeSpecName "kube-api-access-pphcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:44.341719 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.341650    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca" pod "b5881263-63ac-4c33-866f-1f4add3504c6" (UID: "b5881263-63ac-4c33-866f-1f4add3504c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:44.506708 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.492663    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:05:44.506708 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.492722    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5881263-63ac-4c33-866f-1f4add3504c6-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:05:44.506708 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:44.492740    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphcb\" (UniqueName: \"kubernetes.io/projected/b5881263-63ac-4c33-866f-1f4add3504c6-kube-api-access-pphcb\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:05:52.443200 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:05:52.443163    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:05:58.663711 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:05:58.661113    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:06:03.137773 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:03.137742    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:07.267631 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.267591    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"]
Oct 30 23:06:07.268263 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.268244    5028 cpu_manager.go:414] "containerMap: removing container" podUID="b5881263-63ac-4c33-866f-1f4add3504c6" containerName="revoke-keys"
Oct 30 23:06:07.268316 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.268266    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5881263-63ac-4c33-866f-1f4add3504c6" containerName="revoke-keys"
Oct 30 23:06:07.268434 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.268421    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5881263-63ac-4c33-866f-1f4add3504c6" containerName="revoke-keys"
Oct 30 23:06:07.272481 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.272453    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.325468 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.325426    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"]
Oct 30 23:06:07.368929 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.368887    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:06:07.369111 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.368965    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:06:07.369111 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.368986    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:06:07.370085 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.370065    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:06:07.383565 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.383537    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.383733 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.383615    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.383847 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.383823    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5wn\" (UniqueName: \"kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.485103 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.485069    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.485285 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.485166    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5wn\" (UniqueName: \"kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.485285 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.485193    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.486006 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.485971    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.487782 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.487757    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.544292 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.544221    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5wn\" (UniqueName: \"kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn\") pod \"maas-api-revoke-sub-3e31a506-cc4vn\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:07.589566 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:07.589523    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:08.007760 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:08.007728    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"]
Oct 30 23:06:08.501061 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:08.501001    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn" event={"ID":"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173","Type":"ContainerStarted","Data":"84a0b5dafdc48a8016c4b2393df7c8c360a823bb9f03612920244094e11d01ca"}
Oct 30 23:06:08.501609 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:08.501087    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn" event={"ID":"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173","Type":"ContainerStarted","Data":"e9b1951f7d7f3db4b4e09e7136cbc3f8da251a3a6890a6535d71187af44a81e6"}
Oct 30 23:06:09.550505 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:09.547517    5028 generic.go:356] "Generic (PLEG): container finished" podID="a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" containerID="84a0b5dafdc48a8016c4b2393df7c8c360a823bb9f03612920244094e11d01ca" exitCode=0
Oct 30 23:06:09.551724 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:09.551594    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn" event={"ID":"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173","Type":"ContainerDied","Data":"84a0b5dafdc48a8016c4b2393df7c8c360a823bb9f03612920244094e11d01ca"}
Oct 30 23:06:10.809438 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:10.809414    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 23:06:10.836023 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:10.835936    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 23:06:10.918470 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:10.918443    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:11.023305 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.023270    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn\" (UniqueName: \"kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn\") pod \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") "
Oct 30 23:06:11.023493 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.023315    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca\") pod \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") "
Oct 30 23:06:11.023493 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.023337    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca\") pod \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\" (UID: \"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173\") "
Oct 30 23:06:11.023782 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.023758    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca" pod "a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" (UID: "a474e8d2-c5b4-44a6-a5f3-e4a2feb76173"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:11.025545 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.025521    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca" pod "a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" (UID: "a474e8d2-c5b4-44a6-a5f3-e4a2feb76173"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:11.025720 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.025680    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn" pod "a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" (UID: "a474e8d2-c5b4-44a6-a5f3-e4a2feb76173"). InnerVolumeSpecName "kube-api-access-lk5wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:11.124610 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.124544    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk5wn\" (UniqueName: \"kubernetes.io/projected/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-kube-api-access-lk5wn\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:11.124610 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.124574    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:11.124610 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.124584    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:11.591368 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.591280    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn" event={"ID":"a474e8d2-c5b4-44a6-a5f3-e4a2feb76173","Type":"ContainerDied","Data":"e9b1951f7d7f3db4b4e09e7136cbc3f8da251a3a6890a6535d71187af44a81e6"}
Oct 30 23:06:11.591368 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.591344    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b1951f7d7f3db4b4e09e7136cbc3f8da251a3a6890a6535d71187af44a81e6"
Oct 30 23:06:11.591368 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:11.591295    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"
Oct 30 23:06:20.023560 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:20.023476    5028 scope.go:122] "RemoveContainer" containerID="f3995497b1fa52e99bef22dc876a8894475900407892d811f8d9cdef6f98c129"
Oct 30 23:06:22.333299 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:22.333265    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:06:28.652485 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:06:28.652269    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:06:40.000076 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.000036    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"]
Oct 30 23:06:40.000733 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.000711    5028 cpu_manager.go:414] "containerMap: removing container" podUID="a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" containerName="revoke-keys"
Oct 30 23:06:40.000787 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.000736    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" containerName="revoke-keys"
Oct 30 23:06:40.000903 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.000892    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" containerName="revoke-keys"
Oct 30 23:06:40.006152 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.006121    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.057724 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.057676    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"]
Oct 30 23:06:40.086055 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.086024    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.086225 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.086063    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzz6\" (UniqueName: \"kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.086225 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.086172    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.101822 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.101796    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:06:40.101950 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.101825    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:06:40.101950 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.101849    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\""
Oct 30 23:06:40.103448 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.103429    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:06:40.186935 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.186902    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.187097 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.186947    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzz6\" (UniqueName: \"kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.187097 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.186994    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.187639 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.187615    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.189308 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.189281    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.254159 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.254089    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzz6\" (UniqueName: \"kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6\") pod \"maas-api-revoke-sub-74e8d181-n9fkr\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.321547 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.321516    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:40.693141 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.693111    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"]
Oct 30 23:06:40.694423 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:06:40.694398    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7779d432_9922_4090_8771_a2d602cac79f.slice/crio-bcf99f1a50110fe6d499b26617572c9a593dd022b6eb65fe32525bca495dc35a WatchSource:0}: Error finding container bcf99f1a50110fe6d499b26617572c9a593dd022b6eb65fe32525bca495dc35a: Status 404 returned error can't find the container with id bcf99f1a50110fe6d499b26617572c9a593dd022b6eb65fe32525bca495dc35a
Oct 30 23:06:40.869128 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:40.869095    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr" event={"ID":"7779d432-9922-4090-8771-a2d602cac79f","Type":"ContainerStarted","Data":"bcf99f1a50110fe6d499b26617572c9a593dd022b6eb65fe32525bca495dc35a"}
Oct 30 23:06:41.876657 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:41.876572    5028 generic.go:356] "Generic (PLEG): container finished" podID="7779d432-9922-4090-8771-a2d602cac79f" containerID="5a24eb3145fc2d527732ac011cb1f4b5735c6ca2d1949499417cb74c2d69d2d7" exitCode=0
Oct 30 23:06:41.876657 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:41.876648    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr" event={"ID":"7779d432-9922-4090-8771-a2d602cac79f","Type":"ContainerDied","Data":"5a24eb3145fc2d527732ac011cb1f4b5735c6ca2d1949499417cb74c2d69d2d7"}
Oct 30 23:06:43.024161 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.024134    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:43.115049 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.115019    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6\" (UniqueName: \"kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6\") pod \"7779d432-9922-4090-8771-a2d602cac79f\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") "
Oct 30 23:06:43.115236 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.115113    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca\") pod \"7779d432-9922-4090-8771-a2d602cac79f\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") "
Oct 30 23:06:43.115236 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.115208    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca\") pod \"7779d432-9922-4090-8771-a2d602cac79f\" (UID: \"7779d432-9922-4090-8771-a2d602cac79f\") "
Oct 30 23:06:43.115537 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.115503    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca" pod "7779d432-9922-4090-8771-a2d602cac79f" (UID: "7779d432-9922-4090-8771-a2d602cac79f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:43.117275 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.117251    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca" pod "7779d432-9922-4090-8771-a2d602cac79f" (UID: "7779d432-9922-4090-8771-a2d602cac79f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:43.117358 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.117328    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6" pod "7779d432-9922-4090-8771-a2d602cac79f" (UID: "7779d432-9922-4090-8771-a2d602cac79f"). InnerVolumeSpecName "kube-api-access-7xzz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:43.216086 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.216059    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7779d432-9922-4090-8771-a2d602cac79f-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:43.216086 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.216088    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzz6\" (UniqueName: \"kubernetes.io/projected/7779d432-9922-4090-8771-a2d602cac79f-kube-api-access-7xzz6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:43.216292 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.216098    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7779d432-9922-4090-8771-a2d602cac79f-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:43.889289 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.889257    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"
Oct 30 23:06:43.889476 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.889289    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr" event={"ID":"7779d432-9922-4090-8771-a2d602cac79f","Type":"ContainerDied","Data":"bcf99f1a50110fe6d499b26617572c9a593dd022b6eb65fe32525bca495dc35a"}
Oct 30 23:06:43.889476 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:43.889337    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf99f1a50110fe6d499b26617572c9a593dd022b6eb65fe32525bca495dc35a"
Oct 30 23:06:48.048744 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:48.048711    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"]
Oct 30 23:06:48.096264 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:48.096221    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zl4wm"]
Oct 30 23:06:49.289671 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:49.289628    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2de7149-0c84-4c4a-bb39-37dd044f14af" path="/var/lib/kubelet/pods/e2de7149-0c84-4c4a-bb39-37dd044f14af/volumes"
Oct 30 23:06:58.435876 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:06:58.435836    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:06:59.888255 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:06:59.888220    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:00.420936 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.420898    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"]
Oct 30 23:07:00.421439 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.421417    5028 cpu_manager.go:414] "containerMap: removing container" podUID="7779d432-9922-4090-8771-a2d602cac79f" containerName="revoke-keys"
Oct 30 23:07:00.421439 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.421435    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7779d432-9922-4090-8771-a2d602cac79f" containerName="revoke-keys"
Oct 30 23:07:00.421603 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.421514    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="7779d432-9922-4090-8771-a2d602cac79f" containerName="revoke-keys"
Oct 30 23:07:00.425273 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.425254    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.477949 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.477908    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6e41e55b-openshift-default-dockercfg-s6jcz\""
Oct 30 23:07:00.485327 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485145    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.485327 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485208    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl6m\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.485327 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485247    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.485327 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485298    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.486022 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485344    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.486022 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485392    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.486022 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485418    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.486022 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485442    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.486022 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.485481    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.526926 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.526889    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"]
Oct 30 23:07:00.537306 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.537266    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:00.586057 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586019    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl6m\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586057 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586060    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586102    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586130    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586168    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586194    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586217    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586256    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586735 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586347    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586735 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586580    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586735 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586678    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586898 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586763    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.586898 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.586843    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.587128 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.587105    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.588648 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.588613    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.589049 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.589033    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.645316 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.645262    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.645466 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.645351    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl6m\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m\") pod \"e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") " pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.739294 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.739257    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:00.882709 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:00.882661    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:01.168721 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:01.168671    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"]
Oct 30 23:07:01.169929 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:07:01.169902    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec75541_a5e5_4c65_abd4_938be58306d9.slice/crio-fb96e0b8d3871e8d8cec850b7622b3214b3ad40f2298ce7290f261f4f9473a81 WatchSource:0}: Error finding container fb96e0b8d3871e8d8cec850b7622b3214b3ad40f2298ce7290f261f4f9473a81: Status 404 returned error can't find the container with id fb96e0b8d3871e8d8cec850b7622b3214b3ad40f2298ce7290f261f4f9473a81
Oct 30 23:07:01.172675 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:01.172640    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:07:01.172831 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:01.172733    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:07:01.172831 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:01.172766    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:07:01.995655 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:01.995617    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" event={"ID":"9ec75541-a5e5-4c65-abd4-938be58306d9","Type":"ContainerStarted","Data":"84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"}
Oct 30 23:07:01.995860 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:01.995668    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" event={"ID":"9ec75541-a5e5-4c65-abd4-938be58306d9","Type":"ContainerStarted","Data":"fb96e0b8d3871e8d8cec850b7622b3214b3ad40f2298ce7290f261f4f9473a81"}
Oct 30 23:07:02.161308 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:02.161251    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podStartSLOduration=2.161236218 podStartE2EDuration="2.161236218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:02.159541181 +0000 UTC m=+3953.009054177" watchObservedRunningTime="2026-10-30 23:07:02.161236218 +0000 UTC m=+3953.010749191"
Oct 30 23:07:02.739903 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:02.739862    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:02.745147 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:02.745121    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:03.001544 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:03.001455    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:03.002610 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:03.002588    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:03.982861 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:03.982819    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:04.141041 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:04.141005    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:04.293653 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:04.293571    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:04.498447 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:04.498409    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:04.657016 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:04.656914    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:07.439980 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:07.439949    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:07.614045 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:07.614009    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:07.871552 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:07.871517    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:08.024788 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.024748    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"]
Oct 30 23:07:08.029187 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.029163    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.060643 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.060612    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:08.082078 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.080814    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6e41e55b-dockercfg-mqv6d\""
Oct 30 23:07:08.087752 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.087603    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"]
Oct 30 23:07:08.128246 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.128164    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6e41e55b\""
Oct 30 23:07:08.166944 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.166913    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ctl\" (UniqueName: \"kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl\") pod \"payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") " pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.167122 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.166982    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") " pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.226352 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.226314    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:08.268703 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.268659    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ctl\" (UniqueName: \"kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl\") pod \"payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") " pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.268887 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.268801    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") " pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.269513 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.269492    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") " pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.328121 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.328090    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ctl\" (UniqueName: \"kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl\") pod \"payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") " pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.343283 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.343239    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:08.710369 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:08.710337    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"]
Oct 30 23:07:08.711521 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:07:08.711493    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d57e19_1be6_460e_9077_c9d308f5e730.slice/crio-9deb76e0a0a687ee7384dee7ecf62bdee8f25f05eaf23f53f6baccf0f8e9b6b6 WatchSource:0}: Error finding container 9deb76e0a0a687ee7384dee7ecf62bdee8f25f05eaf23f53f6baccf0f8e9b6b6: Status 404 returned error can't find the container with id 9deb76e0a0a687ee7384dee7ecf62bdee8f25f05eaf23f53f6baccf0f8e9b6b6
Oct 30 23:07:09.037600 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:09.037505    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc" event={"ID":"e5d57e19-1be6-460e-9077-c9d308f5e730","Type":"ContainerStarted","Data":"05279f7866edc65d87a1f86fd62187c05a54649d8990753ea95baead4e1c1071"}
Oct 30 23:07:09.037600 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:09.037569    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc" event={"ID":"e5d57e19-1be6-460e-9077-c9d308f5e730","Type":"ContainerStarted","Data":"9deb76e0a0a687ee7384dee7ecf62bdee8f25f05eaf23f53f6baccf0f8e9b6b6"}
Oct 30 23:07:09.037872 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:09.037733    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:09.154864 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:09.154808    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc" podStartSLOduration=2.154792446 podStartE2EDuration="2.154792446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:09.153786103 +0000 UTC m=+3960.003299078" watchObservedRunningTime="2026-10-30 23:07:09.154792446 +0000 UTC m=+3960.004305419"
Oct 30 23:07:12.940611 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:12.940571    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:17.581148 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:17.581088    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:18.058772 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:18.058733    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"]
Oct 30 23:07:18.104354 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:18.104321    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-559hw"]
Oct 30 23:07:19.291680 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:19.291644    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2547f9-8add-4d73-b248-b1068f480d0b" path="/var/lib/kubelet/pods/5c2547f9-8add-4d73-b248-b1068f480d0b/volumes"
Oct 30 23:07:20.048302 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:20.048274    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:22.234949 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:22.234909    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:22.639892 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:22.639812    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:25.137741 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:25.137639    5028 generic.go:356] "Generic (PLEG): container finished" podID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerID="84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e" exitCode=0
Oct 30 23:07:25.137741 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:25.137714    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" event={"ID":"9ec75541-a5e5-4c65-abd4-938be58306d9","Type":"ContainerDied","Data":"84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"}
Oct 30 23:07:25.138218 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:25.138132    5028 scope.go:122] "RemoveContainer" containerID="84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"
Oct 30 23:07:25.139346 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:25.139310    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:07:25.139438 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:25.139390    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:07:25.139438 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:25.139422    5028 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 23:07:26.149265 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:26.149237    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" event={"ID":"9ec75541-a5e5-4c65-abd4-938be58306d9","Type":"ContainerStarted","Data":"4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884"}
Oct 30 23:07:26.740711 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:26.740154    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:26.740711 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:26.740288    5028 patch_prober.go:29] interesting pod/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 23:07:26.740711 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:26.740329    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 23:07:27.043058 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:27.042979    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:27.744767 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:27.741144    5028 patch_prober.go:29] interesting pod/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 23:07:27.744767 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:27.741208    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 23:07:28.456849 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:07:28.456804    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:07:28.742143 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:28.742096    5028 patch_prober.go:29] interesting pod/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 23:07:28.742369 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:28.742180    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 23:07:29.742167 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:29.742128    5028 patch_prober.go:29] interesting pod/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 23:07:29.742738 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:29.742208    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 23:07:30.742555 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:30.742224    5028 patch_prober.go:29] interesting pod/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 23:07:30.742555 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:30.742298    5028 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 23:07:31.745314 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:31.745280    5028 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:31.745766 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:31.745731    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:31.746398 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:31.746382    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:07:31.866402 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:31.866363    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:35.634782 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.634744    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"]
Oct 30 23:07:35.638947 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.638923    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.737568 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.737534    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:07:35.737799 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.737663    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:07:35.737799 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.737743    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:07:35.737799 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.737745    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d5msm\""
Oct 30 23:07:35.740784 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.740765    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"]
Oct 30 23:07:35.769628 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.769594    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.769628 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.769634    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxpk\" (UniqueName: \"kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.769891 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.769795    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.870550 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.870509    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.870550 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.870553    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxpk\" (UniqueName: \"kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.870824 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.870599    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.871225 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.871193    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.873027 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.872999    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.930305 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.930211    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxpk\" (UniqueName: \"kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk\") pod \"maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:35.951805 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:35.951775    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:36.322885 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:36.322854    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"]
Oct 30 23:07:36.323308 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:07:36.323281    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bdbd7a_8ccc_4f0f_abc6_037da4d04375.slice/crio-d9881e997a0deaeed7bf4909dfe88c9b5bb1bb0516bc6578d0e5e0fc876d6180 WatchSource:0}: Error finding container d9881e997a0deaeed7bf4909dfe88c9b5bb1bb0516bc6578d0e5e0fc876d6180: Status 404 returned error can't find the container with id d9881e997a0deaeed7bf4909dfe88c9b5bb1bb0516bc6578d0e5e0fc876d6180
Oct 30 23:07:36.452566 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:36.452533    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:37.218457 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:37.218420    5028 generic.go:356] "Generic (PLEG): container finished" podID="37bdbd7a-8ccc-4f0f-abc6-037da4d04375" containerID="c9d743a9d6d7996c68cc9a8b7b78477540eb4ac827253f28b1a1b86058c592ba" exitCode=0
Oct 30 23:07:37.219066 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:37.218480    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6" event={"ID":"37bdbd7a-8ccc-4f0f-abc6-037da4d04375","Type":"ContainerDied","Data":"c9d743a9d6d7996c68cc9a8b7b78477540eb4ac827253f28b1a1b86058c592ba"}
Oct 30 23:07:37.219066 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:37.218516    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6" event={"ID":"37bdbd7a-8ccc-4f0f-abc6-037da4d04375","Type":"ContainerStarted","Data":"d9881e997a0deaeed7bf4909dfe88c9b5bb1bb0516bc6578d0e5e0fc876d6180"}
Oct 30 23:07:38.367745 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.367719    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:38.497287 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.497206    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca\") pod \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") "
Oct 30 23:07:38.497449 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.497347    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk\" (UniqueName: \"kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk\") pod \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") "
Oct 30 23:07:38.497449 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.497435    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca\") pod \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\" (UID: \"37bdbd7a-8ccc-4f0f-abc6-037da4d04375\") "
Oct 30 23:07:38.497662 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.497641    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca" pod "37bdbd7a-8ccc-4f0f-abc6-037da4d04375" (UID: "37bdbd7a-8ccc-4f0f-abc6-037da4d04375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:07:38.497854 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.497829    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:07:38.499580 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.499551    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca" pod "37bdbd7a-8ccc-4f0f-abc6-037da4d04375" (UID: "37bdbd7a-8ccc-4f0f-abc6-037da4d04375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:07:38.499773 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.499747    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk" pod "37bdbd7a-8ccc-4f0f-abc6-037da4d04375" (UID: "37bdbd7a-8ccc-4f0f-abc6-037da4d04375"). InnerVolumeSpecName "kube-api-access-htxpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:07:38.598743 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.598682    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxpk\" (UniqueName: \"kubernetes.io/projected/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-kube-api-access-htxpk\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:07:38.598743 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:38.598741    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bdbd7a-8ccc-4f0f-abc6-037da4d04375-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:07:39.231738 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:39.231704    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"
Oct 30 23:07:39.231738 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:39.231723    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6" event={"ID":"37bdbd7a-8ccc-4f0f-abc6-037da4d04375","Type":"ContainerDied","Data":"d9881e997a0deaeed7bf4909dfe88c9b5bb1bb0516bc6578d0e5e0fc876d6180"}
Oct 30 23:07:39.232016 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:39.231776    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9881e997a0deaeed7bf4909dfe88c9b5bb1bb0516bc6578d0e5e0fc876d6180"
Oct 30 23:07:40.944063 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:40.944027    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:45.541183 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:45.541145    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:47.133566 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:47.133525    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"]
Oct 30 23:07:47.562477 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:47.562430    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:47.728595 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:47.728562    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:50.045728 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:50.045614    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:52.134576 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.134524    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc" podUID="e5d57e19-1be6-460e-9077-c9d308f5e730" containerName="payload-processing" containerID="cri-o://05279f7866edc65d87a1f86fd62187c05a54649d8990753ea95baead4e1c1071" gracePeriod=25
Oct 30 23:07:52.317845 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.317806    5028 generic.go:356] "Generic (PLEG): container finished" podID="e5d57e19-1be6-460e-9077-c9d308f5e730" containerID="05279f7866edc65d87a1f86fd62187c05a54649d8990753ea95baead4e1c1071" exitCode=0
Oct 30 23:07:52.318028 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.317874    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc" event={"ID":"e5d57e19-1be6-460e-9077-c9d308f5e730","Type":"ContainerDied","Data":"05279f7866edc65d87a1f86fd62187c05a54649d8990753ea95baead4e1c1071"}
Oct 30 23:07:52.390115 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.390050    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:52.544143 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.544098    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl\" (UniqueName: \"kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl\") pod \"e5d57e19-1be6-460e-9077-c9d308f5e730\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") "
Oct 30 23:07:52.544363 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.544188    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume\") pod \"e5d57e19-1be6-460e-9077-c9d308f5e730\" (UID: \"e5d57e19-1be6-460e-9077-c9d308f5e730\") "
Oct 30 23:07:52.544565 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.544539    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume" pod "e5d57e19-1be6-460e-9077-c9d308f5e730" (UID: "e5d57e19-1be6-460e-9077-c9d308f5e730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:07:52.546301 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.546277    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl" pod "e5d57e19-1be6-460e-9077-c9d308f5e730" (UID: "e5d57e19-1be6-460e-9077-c9d308f5e730"). InnerVolumeSpecName "kube-api-access-85ctl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:07:52.645085 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.645007    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ctl\" (UniqueName: \"kubernetes.io/projected/e5d57e19-1be6-460e-9077-c9d308f5e730-kube-api-access-85ctl\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:07:52.645085 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:52.645033    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d57e19-1be6-460e-9077-c9d308f5e730-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:07:53.324671 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:53.324640    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"
Oct 30 23:07:53.324671 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:53.324662    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc" event={"ID":"e5d57e19-1be6-460e-9077-c9d308f5e730","Type":"ContainerDied","Data":"9deb76e0a0a687ee7384dee7ecf62bdee8f25f05eaf23f53f6baccf0f8e9b6b6"}
Oct 30 23:07:53.325214 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:53.324744    5028 scope.go:122] "RemoveContainer" containerID="05279f7866edc65d87a1f86fd62187c05a54649d8990753ea95baead4e1c1071"
Oct 30 23:07:53.484186 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:53.484152    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"]
Oct 30 23:07:53.531200 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:53.531167    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e41e55b-77fbff758f-7xjhc"]
Oct 30 23:07:54.032634 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:54.032594    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:55.289645 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:55.289612    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d57e19-1be6-460e-9077-c9d308f5e730" path="/var/lib/kubelet/pods/e5d57e19-1be6-460e-9077-c9d308f5e730/volumes"
Oct 30 23:07:55.442619 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:55.442583    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:55.708074 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:55.708045    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:55.900302 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:55.900267    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"]
Oct 30 23:07:55.948249 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:55.948213    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e41e55b-zr9z6"]
Oct 30 23:07:56.826439 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:56.826404    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"]
Oct 30 23:07:56.826874 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:56.826645    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" containerID="cri-o://4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884" gracePeriod=30
Oct 30 23:07:56.834574 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:56.834549    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:57.291536 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:57.291504    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bdbd7a-8ccc-4f0f-abc6-037da4d04375" path="/var/lib/kubelet/pods/37bdbd7a-8ccc-4f0f-abc6-037da4d04375/volumes"
Oct 30 23:07:58.116357 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:58.116297    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:58.435740 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:07:58.435646    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:07:58.723703 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:58.723659    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:07:58.993980 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:07:58.993889    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:01.747756 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:01.747654    5028 patch_prober.go:29] interesting pod/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:08:01.747756 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:01.747737    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:08:02.098839 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.098809    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:08:02.137697 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137641    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.137868 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137724    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.137868 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137767    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.137868 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137812    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.137868 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137850    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.138101 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137878    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.138101 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.137995    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.138101 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138030    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.138101 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138059    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs\") pod \"9ec75541-a5e5-4c65-abd4-938be58306d9\" (UID: \"9ec75541-a5e5-4c65-abd4-938be58306d9\") "
Oct 30 23:08:02.138333 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138080    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:02.138333 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138205    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:08:02.138458 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138325    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:02.138458 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138434    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:02.138576 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138474    5028 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.138576 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138493    5028 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec75541-a5e5-4c65-abd4-938be58306d9-istiod-ca-cert\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.138576 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138508    5028 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-credential-socket\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.138809 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.138781    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:02.140455 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.140413    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:02.140592 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.140515    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:08:02.140859 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.140833    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "kube-api-access-ldl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:02.140986 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.140866    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token" pod "9ec75541-a5e5-4c65-abd4-938be58306d9" (UID: "9ec75541-a5e5-4c65-abd4-938be58306d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:02.239647 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.239597    5028 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-token\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.239647 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.239638    5028 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-podinfo\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.239647 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.239656    5028 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-data\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.239994 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.239672    5028 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-istio-envoy\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.239994 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.239705    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldl6m\" (UniqueName: \"kubernetes.io/projected/9ec75541-a5e5-4c65-abd4-938be58306d9-kube-api-access-ldl6m\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.239994 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.239721    5028 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec75541-a5e5-4c65-abd4-938be58306d9-workload-certs\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:02.384084 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.383996    5028 generic.go:356] "Generic (PLEG): container finished" podID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerID="4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884" exitCode=0
Oct 30 23:08:02.384084 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.384039    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"
Oct 30 23:08:02.384274 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.384082    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" event={"ID":"9ec75541-a5e5-4c65-abd4-938be58306d9","Type":"ContainerDied","Data":"4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884"}
Oct 30 23:08:02.384274 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.384134    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn" event={"ID":"9ec75541-a5e5-4c65-abd4-938be58306d9","Type":"ContainerDied","Data":"fb96e0b8d3871e8d8cec850b7622b3214b3ad40f2298ce7290f261f4f9473a81"}
Oct 30 23:08:02.384274 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.384155    5028 scope.go:122] "RemoveContainer" containerID="4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884"
Oct 30 23:08:02.394260 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.394240    5028 scope.go:122] "RemoveContainer" containerID="84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"
Oct 30 23:08:02.403264 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.403246    5028 scope.go:122] "RemoveContainer" containerID="4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884"
Oct 30 23:08:02.403539 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:08:02.403520    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884\": container with ID starting with 4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884 not found: ID does not exist" containerID="4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884"
Oct 30 23:08:02.403583 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.403550    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884"} err="failed to get container status \"4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884\": rpc error: code = NotFound desc = could not find container \"4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884\": container with ID starting with 4a22c78c087fe51d688d9094bd20279db59d6ab174ed175d2e6f27c49a6d5884 not found: ID does not exist"
Oct 30 23:08:02.403583 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.403571    5028 scope.go:122] "RemoveContainer" containerID="84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"
Oct 30 23:08:02.403825 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:08:02.403808    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e\": container with ID starting with 84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e not found: ID does not exist" containerID="84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"
Oct 30 23:08:02.403879 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.403830    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e"} err="failed to get container status \"84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e\": rpc error: code = NotFound desc = could not find container \"84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e\": container with ID starting with 84df097a88462d02b9cf3b6a305508086f8670b4f8efa60e7ed4cd9b8f90660e not found: ID does not exist"
Oct 30 23:08:02.544714 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.544652    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"]
Oct 30 23:08:02.547056 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:02.547029    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6e41e55b-openshift-default-789db867c9-x2kzn"]
Oct 30 23:08:03.290591 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:03.290558    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" path="/var/lib/kubelet/pods/9ec75541-a5e5-4c65-abd4-938be58306d9/volumes"
Oct 30 23:08:03.297620 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:03.297593    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:03.517283 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:03.517250    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:03.673959 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:03.673873    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:06.423673 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.423638    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:06.588706 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.588663    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:06.831894 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.831859    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"]
Oct 30 23:08:06.832297 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832285    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy"
Oct 30 23:08:06.832351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832299    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy"
Oct 30 23:08:06.832351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832310    5028 cpu_manager.go:414] "containerMap: removing container" podUID="e5d57e19-1be6-460e-9077-c9d308f5e730" containerName="payload-processing"
Oct 30 23:08:06.832351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832315    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d57e19-1be6-460e-9077-c9d308f5e730" containerName="payload-processing"
Oct 30 23:08:06.832351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832340    5028 cpu_manager.go:414] "containerMap: removing container" podUID="37bdbd7a-8ccc-4f0f-abc6-037da4d04375" containerName="revoke-keys"
Oct 30 23:08:06.832351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832346    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bdbd7a-8ccc-4f0f-abc6-037da4d04375" containerName="revoke-keys"
Oct 30 23:08:06.832521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832362    5028 cpu_manager.go:414] "containerMap: removing container" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy"
Oct 30 23:08:06.832521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832367    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy"
Oct 30 23:08:06.832521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832448    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy"
Oct 30 23:08:06.832521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832456    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec75541-a5e5-4c65-abd4-938be58306d9" containerName="istio-proxy"
Oct 30 23:08:06.832521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832462    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d57e19-1be6-460e-9077-c9d308f5e730" containerName="payload-processing"
Oct 30 23:08:06.832521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.832473    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bdbd7a-8ccc-4f0f-abc6-037da4d04375" containerName="revoke-keys"
Oct 30 23:08:06.836799 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.836783    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:06.887362 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.887325    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvb2n\" (UniqueName: \"kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n\") pod \"payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:06.887634 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.887613    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:06.891754 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.891726    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"]
Oct 30 23:08:06.934661 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.934624    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-152666\""
Oct 30 23:08:06.934977 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.934951    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-152666-dockercfg-szxzn\""
Oct 30 23:08:06.987004 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.986967    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"]
Oct 30 23:08:06.988811 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.988791    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvb2n\" (UniqueName: \"kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n\") pod \"payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:06.988902 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.988827    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:06.989518 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.989496    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:06.991021 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:06.991004    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.050409 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.050374    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"]
Oct 30 23:08:07.055599 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.055566    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvb2n\" (UniqueName: \"kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n\") pod \"payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:07.090192 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.090115    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drp7q\" (UniqueName: \"kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q\") pod \"payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.090192 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.090153    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.150121 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.150087    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:07.191952 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.191920    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drp7q\" (UniqueName: \"kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q\") pod \"payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.192144 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.191961    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.192945 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.192920    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.252125 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.252088    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drp7q\" (UniqueName: \"kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q\") pod \"payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.304648 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.304616    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:07.517697 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.517653    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"]
Oct 30 23:08:07.518372 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:08:07.518348    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b975ea_d988_427d_9ae7_5aa9b674fd65.slice/crio-4e5b663b35d37047d3574cd85fdcf3bc8f4f3f374949147e6c9ff5e4a32a5b1b WatchSource:0}: Error finding container 4e5b663b35d37047d3574cd85fdcf3bc8f4f3f374949147e6c9ff5e4a32a5b1b: Status 404 returned error can't find the container with id 4e5b663b35d37047d3574cd85fdcf3bc8f4f3f374949147e6c9ff5e4a32a5b1b
Oct 30 23:08:07.679953 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:07.679923    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"]
Oct 30 23:08:07.680938 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:08:07.680912    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5743d643_8f49_4d28_a053_baf268732a51.slice/crio-bea2ba50b861dc9829282fe2a906518175c304e4ada63976d8f2b7e4c6e5bb86 WatchSource:0}: Error finding container bea2ba50b861dc9829282fe2a906518175c304e4ada63976d8f2b7e4c6e5bb86: Status 404 returned error can't find the container with id bea2ba50b861dc9829282fe2a906518175c304e4ada63976d8f2b7e4c6e5bb86
Oct 30 23:08:08.422363 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.422321    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4" event={"ID":"13b975ea-d988-427d-9ae7-5aa9b674fd65","Type":"ContainerStarted","Data":"bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902"}
Oct 30 23:08:08.422595 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.422386    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4" event={"ID":"13b975ea-d988-427d-9ae7-5aa9b674fd65","Type":"ContainerStarted","Data":"4e5b663b35d37047d3574cd85fdcf3bc8f4f3f374949147e6c9ff5e4a32a5b1b"}
Oct 30 23:08:08.422595 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.422414    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:08.424027 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.423993    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp" event={"ID":"5743d643-8f49-4d28-a053-baf268732a51","Type":"ContainerStarted","Data":"7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b"}
Oct 30 23:08:08.424145 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.424047    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp" event={"ID":"5743d643-8f49-4d28-a053-baf268732a51","Type":"ContainerStarted","Data":"bea2ba50b861dc9829282fe2a906518175c304e4ada63976d8f2b7e4c6e5bb86"}
Oct 30 23:08:08.424145 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.424086    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:08.454967 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.454933    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:08.604228 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.604184    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:08.658448 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.658392    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4" podStartSLOduration=2.658378005 podStartE2EDuration="2.658378005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:08.543530946 +0000 UTC m=+4019.393043919" watchObservedRunningTime="2026-10-30 23:08:08.658378005 +0000 UTC m=+4019.507891041"
Oct 30 23:08:08.658657 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:08.658493    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp" podStartSLOduration=2.658486808 podStartE2EDuration="2.658486808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:08.65537276 +0000 UTC m=+4019.504885760" watchObservedRunningTime="2026-10-30 23:08:08.658486808 +0000 UTC m=+4019.507999783"
Oct 30 23:08:09.514912 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:09.514869    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:09.669902 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:09.669869    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:09.824993 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:09.824897    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:14.245191 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:14.245159    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:18.848739 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:18.848701    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:19.433141 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:19.433110    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:08:19.433457 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:19.433437    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:08:20.058338 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:20.058305    5028 scope.go:122] "RemoveContainer" containerID="fc959924b6b08965d1f1024686e061a2f24605a97e6d0e6f49b358e98563051c"
Oct 30 23:08:20.067934 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:20.067911    5028 scope.go:122] "RemoveContainer" containerID="c407ec172e2e7b5a0da7d67b2254e805005c23b4e84d3a0d3e707ecde04e1534"
Oct 30 23:08:23.004513 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:23.004476    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:23.474588 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:23.474545    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:24.863622 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:24.863589    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:28.228289 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:28.228239    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:28.389264 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:28.389228    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:28.439353 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:08:28.439318    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:08:33.040490 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:33.040453    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:37.636486 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:37.636451    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:42.146680 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:42.146640    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:46.734036 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:46.733967    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:50.057108 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:50.057074    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"]
Oct 30 23:08:50.104304 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:50.104266    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89007d93-7cd64"]
Oct 30 23:08:51.243754 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:51.243718    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:51.292735 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:51.292697    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae3bdd5-b488-410f-b539-064d3e4d212f" path="/var/lib/kubelet/pods/aae3bdd5-b488-410f-b539-064d3e4d212f/volumes"
Oct 30 23:08:55.828512 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:08:55.828476    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:08:58.435308 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:08:58.435271    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:09:00.364086 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:00.364045    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:05.045800 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:05.045770    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:09.557667 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:09.557635    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:10.824916 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:10.824871    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:14.145005 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:14.144961    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:18.642638 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:18.642600    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:23.141850 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:23.141765    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:23.320026 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:23.319993    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:27.636200 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:27.636158    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:28.435649 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:09:28.435608    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:09:32.238904 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:32.238868    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:36.831985 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:36.831948    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:41.359939 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:41.359899    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:45.842237 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:45.842205    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:48.932885 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:48.932844    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"]
Oct 30 23:09:48.936477 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:48.936460    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:48.988624 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:48.988572    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:09:48.991650 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:48.991624    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"]
Oct 30 23:09:49.034961 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.034931    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:09:49.034961 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.034951    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:09:49.035196 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.035025    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d5msm\""
Oct 30 23:09:49.133618 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.133579    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.133825 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.133755    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.133876 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.133840    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6fv\" (UniqueName: \"kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.234789 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.234742    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.234969 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.234838    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6fv\" (UniqueName: \"kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.234969 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.234917    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.235460 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.235430    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.237240 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.237217    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.293852 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.293819    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6fv\" (UniqueName: \"kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv\") pod \"maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.549927 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.549846    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:49.922320 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:49.922287    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"]
Oct 30 23:09:49.923331 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:09:49.923302    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6559ca9_0344_435f_9685_e4643ed4d49b.slice/crio-920b066958afbb63d159abebd19c107209692c7bf153e7e736c63039b72c1025 WatchSource:0}: Error finding container 920b066958afbb63d159abebd19c107209692c7bf153e7e736c63039b72c1025: Status 404 returned error can't find the container with id 920b066958afbb63d159abebd19c107209692c7bf153e7e736c63039b72c1025
Oct 30 23:09:50.042110 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:50.042077    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99" event={"ID":"f6559ca9-0344-435f-9685-e4643ed4d49b","Type":"ContainerStarted","Data":"920b066958afbb63d159abebd19c107209692c7bf153e7e736c63039b72c1025"}
Oct 30 23:09:50.575448 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:50.575374    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:51.050417 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:51.050381    5028 generic.go:356] "Generic (PLEG): container finished" podID="f6559ca9-0344-435f-9685-e4643ed4d49b" containerID="b8dc94532b43e3ea9f087e598d30d0f8308defc393c16dae38edf73134bcca98" exitCode=0
Oct 30 23:09:51.050417 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:51.050420    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99" event={"ID":"f6559ca9-0344-435f-9685-e4643ed4d49b","Type":"ContainerDied","Data":"b8dc94532b43e3ea9f087e598d30d0f8308defc393c16dae38edf73134bcca98"}
Oct 30 23:09:52.194415 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.194389    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:52.263521 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.263487    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca\") pod \"f6559ca9-0344-435f-9685-e4643ed4d49b\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") "
Oct 30 23:09:52.263737 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.263718    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv\" (UniqueName: \"kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv\") pod \"f6559ca9-0344-435f-9685-e4643ed4d49b\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") "
Oct 30 23:09:52.263823 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.263806    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca\") pod \"f6559ca9-0344-435f-9685-e4643ed4d49b\" (UID: \"f6559ca9-0344-435f-9685-e4643ed4d49b\") "
Oct 30 23:09:52.263925 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.263905    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca" pod "f6559ca9-0344-435f-9685-e4643ed4d49b" (UID: "f6559ca9-0344-435f-9685-e4643ed4d49b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:09:52.264104 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.264089    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:09:52.265798 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.265772    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca" pod "f6559ca9-0344-435f-9685-e4643ed4d49b" (UID: "f6559ca9-0344-435f-9685-e4643ed4d49b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:09:52.265904 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.265882    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv" pod "f6559ca9-0344-435f-9685-e4643ed4d49b" (UID: "f6559ca9-0344-435f-9685-e4643ed4d49b"). InnerVolumeSpecName "kube-api-access-pv6fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:09:52.365326 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.365239    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6559ca9-0344-435f-9685-e4643ed4d49b-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:09:52.365326 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:52.365283    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv6fv\" (UniqueName: \"kubernetes.io/projected/f6559ca9-0344-435f-9685-e4643ed4d49b-kube-api-access-pv6fv\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:09:53.064155 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:53.064122    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"
Oct 30 23:09:53.064353 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:53.064152    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99" event={"ID":"f6559ca9-0344-435f-9685-e4643ed4d49b","Type":"ContainerDied","Data":"920b066958afbb63d159abebd19c107209692c7bf153e7e736c63039b72c1025"}
Oct 30 23:09:53.064353 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:53.064203    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="920b066958afbb63d159abebd19c107209692c7bf153e7e736c63039b72c1025"
Oct 30 23:09:55.343713 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:09:55.343655    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:09:58.435486 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:09:58.435449    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:10:00.321546 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:00.321504    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"]
Oct 30 23:10:00.423794 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:00.423755    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"]
Oct 30 23:10:00.678337 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:00.678253    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:00.846609 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:00.846574    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:05.321877 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.321828    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp" podUID="5743d643-8f49-4d28-a053-baf268732a51" containerName="payload-processing" containerID="cri-o://7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b" gracePeriod=25
Oct 30 23:10:05.424380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.424331    5028 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4" podUID="13b975ea-d988-427d-9ae7-5aa9b674fd65" containerName="payload-pre-processing" containerID="cri-o://bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902" gracePeriod=25
Oct 30 23:10:05.585026 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.584962    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:10:05.694511 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.694486    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:10:05.697983 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.697958    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume\") pod \"5743d643-8f49-4d28-a053-baf268732a51\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") "
Oct 30 23:10:05.698138 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.698026    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q\" (UniqueName: \"kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q\") pod \"5743d643-8f49-4d28-a053-baf268732a51\" (UID: \"5743d643-8f49-4d28-a053-baf268732a51\") "
Oct 30 23:10:05.698415 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.698388    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume" pod "5743d643-8f49-4d28-a053-baf268732a51" (UID: "5743d643-8f49-4d28-a053-baf268732a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:05.700483 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.700460    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q" pod "5743d643-8f49-4d28-a053-baf268732a51" (UID: "5743d643-8f49-4d28-a053-baf268732a51"). InnerVolumeSpecName "kube-api-access-drp7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:05.799248 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.799212    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n\" (UniqueName: \"kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n\") pod \"13b975ea-d988-427d-9ae7-5aa9b674fd65\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") "
Oct 30 23:10:05.799414 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.799329    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume\") pod \"13b975ea-d988-427d-9ae7-5aa9b674fd65\" (UID: \"13b975ea-d988-427d-9ae7-5aa9b674fd65\") "
Oct 30 23:10:05.799559 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.799545    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5743d643-8f49-4d28-a053-baf268732a51-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:05.799652 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.799562    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drp7q\" (UniqueName: \"kubernetes.io/projected/5743d643-8f49-4d28-a053-baf268732a51-kube-api-access-drp7q\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:05.799792 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.799761    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume" pod "13b975ea-d988-427d-9ae7-5aa9b674fd65" (UID: "13b975ea-d988-427d-9ae7-5aa9b674fd65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:05.801486 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.801466    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n" pod "13b975ea-d988-427d-9ae7-5aa9b674fd65" (UID: "13b975ea-d988-427d-9ae7-5aa9b674fd65"). InnerVolumeSpecName "kube-api-access-xvb2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:05.900895 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.900805    5028 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b975ea-d988-427d-9ae7-5aa9b674fd65-plugins-config-volume\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:05.900895 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:05.900840    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvb2n\" (UniqueName: \"kubernetes.io/projected/13b975ea-d988-427d-9ae7-5aa9b674fd65-kube-api-access-xvb2n\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:06.156626 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.156537    5028 generic.go:356] "Generic (PLEG): container finished" podID="5743d643-8f49-4d28-a053-baf268732a51" containerID="7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b" exitCode=0
Oct 30 23:10:06.156626 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.156582    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"
Oct 30 23:10:06.156851 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.156617    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp" event={"ID":"5743d643-8f49-4d28-a053-baf268732a51","Type":"ContainerDied","Data":"7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b"}
Oct 30 23:10:06.156851 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.156678    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp" event={"ID":"5743d643-8f49-4d28-a053-baf268732a51","Type":"ContainerDied","Data":"bea2ba50b861dc9829282fe2a906518175c304e4ada63976d8f2b7e4c6e5bb86"}
Oct 30 23:10:06.156851 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.156724    5028 scope.go:122] "RemoveContainer" containerID="7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b"
Oct 30 23:10:06.158158 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.158133    5028 generic.go:356] "Generic (PLEG): container finished" podID="13b975ea-d988-427d-9ae7-5aa9b674fd65" containerID="bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902" exitCode=0
Oct 30 23:10:06.158280 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.158180    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"
Oct 30 23:10:06.158280 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.158221    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4" event={"ID":"13b975ea-d988-427d-9ae7-5aa9b674fd65","Type":"ContainerDied","Data":"bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902"}
Oct 30 23:10:06.158382 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.158297    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4" event={"ID":"13b975ea-d988-427d-9ae7-5aa9b674fd65","Type":"ContainerDied","Data":"4e5b663b35d37047d3574cd85fdcf3bc8f4f3f374949147e6c9ff5e4a32a5b1b"}
Oct 30 23:10:06.166874 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.166851    5028 scope.go:122] "RemoveContainer" containerID="7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b"
Oct 30 23:10:06.167173 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:10:06.167153    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b\": container with ID starting with 7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b not found: ID does not exist" containerID="7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b"
Oct 30 23:10:06.167265 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.167181    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b"} err="failed to get container status \"7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b\": rpc error: code = NotFound desc = could not find container \"7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b\": container with ID starting with 7725bb3fa8cb2c0840fdff2f2f2989ef273a931c5746c4d660dc31dd801e511b not found: ID does not exist"
Oct 30 23:10:06.167265 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.167199    5028 scope.go:122] "RemoveContainer" containerID="bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902"
Oct 30 23:10:06.177399 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.177363    5028 scope.go:122] "RemoveContainer" containerID="bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902"
Oct 30 23:10:06.177648 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:10:06.177632    5028 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902\": container with ID starting with bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902 not found: ID does not exist" containerID="bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902"
Oct 30 23:10:06.177729 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.177656    5028 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902"} err="failed to get container status \"bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902\": rpc error: code = NotFound desc = could not find container \"bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902\": container with ID starting with bfbc134da07871919ce0e7d639112c71776dd972de04631251d70332e899a902 not found: ID does not exist"
Oct 30 23:10:06.314558 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.314523    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"]
Oct 30 23:10:06.361867 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.361825    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-152666-5b4bb5865f-2qjwp"]
Oct 30 23:10:06.424207 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.424124    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"]
Oct 30 23:10:06.428532 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:06.428507    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-152666-66b4cc55d7-b7ps4"]
Oct 30 23:10:07.291496 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:07.291466    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b975ea-d988-427d-9ae7-5aa9b674fd65" path="/var/lib/kubelet/pods/13b975ea-d988-427d-9ae7-5aa9b674fd65/volumes"
Oct 30 23:10:07.291888 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:07.291873    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5743d643-8f49-4d28-a053-baf268732a51" path="/var/lib/kubelet/pods/5743d643-8f49-4d28-a053-baf268732a51/volumes"
Oct 30 23:10:08.262157 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:08.262118    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:09.688415 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:09.688375    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:09.968996 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:09.968957    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:10.255765 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:10.255667    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"]
Oct 30 23:10:10.302814 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:10.302779    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-152666-n9k99"]
Oct 30 23:10:10.724481 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:10.724447    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:11.290144 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:11.290113    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6559ca9-0344-435f-9685-e4643ed4d49b" path="/var/lib/kubelet/pods/f6559ca9-0344-435f-9685-e4643ed4d49b/volumes"
Oct 30 23:10:16.993328 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:16.993295    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:17.160924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:17.160889    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:17.377649 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:17.377559    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:20.113236 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:20.113202    5028 scope.go:122] "RemoveContainer" containerID="0726535e3695850ea427cc292b52ec2bc67dd9c7409d5aab044dec54219e44e7"
Oct 30 23:10:20.123676 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:20.123656    5028 scope.go:122] "RemoveContainer" containerID="39b36f89078c93c712559f6f458b943607c03495d33500fded40929d9845cdc5"
Oct 30 23:10:23.675422 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:23.675382    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:28.527128 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:10:28.527079    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:10:44.737527 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:44.737489    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:10:45.051731 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:45.051606    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"]
Oct 30 23:10:45.094878 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:45.094841    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dcdjt"]
Oct 30 23:10:45.290326 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:45.290294    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5881263-63ac-4c33-866f-1f4add3504c6" path="/var/lib/kubelet/pods/b5881263-63ac-4c33-866f-1f4add3504c6/volumes"
Oct 30 23:10:47.894207 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894163    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"]
Oct 30 23:10:47.894814 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894793    5028 cpu_manager.go:414] "containerMap: removing container" podUID="f6559ca9-0344-435f-9685-e4643ed4d49b" containerName="revoke-keys"
Oct 30 23:10:47.894924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894816    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6559ca9-0344-435f-9685-e4643ed4d49b" containerName="revoke-keys"
Oct 30 23:10:47.894924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894834    5028 cpu_manager.go:414] "containerMap: removing container" podUID="13b975ea-d988-427d-9ae7-5aa9b674fd65" containerName="payload-pre-processing"
Oct 30 23:10:47.894924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894841    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b975ea-d988-427d-9ae7-5aa9b674fd65" containerName="payload-pre-processing"
Oct 30 23:10:47.894924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894851    5028 cpu_manager.go:414] "containerMap: removing container" podUID="5743d643-8f49-4d28-a053-baf268732a51" containerName="payload-processing"
Oct 30 23:10:47.894924 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.894881    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5743d643-8f49-4d28-a053-baf268732a51" containerName="payload-processing"
Oct 30 23:10:47.895205 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.895044    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6559ca9-0344-435f-9685-e4643ed4d49b" containerName="revoke-keys"
Oct 30 23:10:47.895205 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.895057    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b975ea-d988-427d-9ae7-5aa9b674fd65" containerName="payload-pre-processing"
Oct 30 23:10:47.895205 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.895069    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="5743d643-8f49-4d28-a053-baf268732a51" containerName="payload-processing"
Oct 30 23:10:47.898274 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.898251    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:47.952558 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.952527    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"]
Oct 30 23:10:47.996856 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.996825    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:10:47.996856 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.996848    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:10:47.996856 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.996840    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:10:47.997162 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:47.996920    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d5msm\""
Oct 30 23:10:48.001335 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.001312    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.001433 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.001348    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.001433 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.001382    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdzr\" (UniqueName: \"kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.102973 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.102939    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.102973 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.102981    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.103206 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.103020    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdzr\" (UniqueName: \"kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.103724 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.103672    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.105356 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.105337    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.161932 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.161859    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdzr\" (UniqueName: \"kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr\") pod \"maas-api-revoke-sub-f03629db-45jpv\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.212059 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.212010    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:48.580945 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.580915    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"]
Oct 30 23:10:48.582222 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:10:48.582195    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734b4d61_4cd9_49d3_95cf_1005c50eea22.slice/crio-e4f31cb64ff0ee52cebdf081e867f470e958510b0f42cc2af77b355cf8a66ccd WatchSource:0}: Error finding container e4f31cb64ff0ee52cebdf081e867f470e958510b0f42cc2af77b355cf8a66ccd: Status 404 returned error can't find the container with id e4f31cb64ff0ee52cebdf081e867f470e958510b0f42cc2af77b355cf8a66ccd
Oct 30 23:10:48.593952 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:48.593928    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv" event={"ID":"734b4d61-4cd9-49d3-95cf-1005c50eea22","Type":"ContainerStarted","Data":"e4f31cb64ff0ee52cebdf081e867f470e958510b0f42cc2af77b355cf8a66ccd"}
Oct 30 23:10:49.602175 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:49.602085    5028 generic.go:356] "Generic (PLEG): container finished" podID="734b4d61-4cd9-49d3-95cf-1005c50eea22" containerID="d32491b323bbe3b503793675e027b313c723801c8db440b569bd27bdd0d9d238" exitCode=0
Oct 30 23:10:49.602575 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:49.602171    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv" event={"ID":"734b4d61-4cd9-49d3-95cf-1005c50eea22","Type":"ContainerDied","Data":"d32491b323bbe3b503793675e027b313c723801c8db440b569bd27bdd0d9d238"}
Oct 30 23:10:50.747561 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.747535    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:50.828996 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.828960    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca\") pod \"734b4d61-4cd9-49d3-95cf-1005c50eea22\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") "
Oct 30 23:10:50.829200 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.829079    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr\" (UniqueName: \"kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr\") pod \"734b4d61-4cd9-49d3-95cf-1005c50eea22\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") "
Oct 30 23:10:50.829200 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.829150    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca\") pod \"734b4d61-4cd9-49d3-95cf-1005c50eea22\" (UID: \"734b4d61-4cd9-49d3-95cf-1005c50eea22\") "
Oct 30 23:10:50.829547 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.829523    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca" pod "734b4d61-4cd9-49d3-95cf-1005c50eea22" (UID: "734b4d61-4cd9-49d3-95cf-1005c50eea22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:50.831247 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.831225    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr" pod "734b4d61-4cd9-49d3-95cf-1005c50eea22" (UID: "734b4d61-4cd9-49d3-95cf-1005c50eea22"). InnerVolumeSpecName "kube-api-access-sxdzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:50.831351 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.831256    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca" pod "734b4d61-4cd9-49d3-95cf-1005c50eea22" (UID: "734b4d61-4cd9-49d3-95cf-1005c50eea22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:10:50.930533 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.930443    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdzr\" (UniqueName: \"kubernetes.io/projected/734b4d61-4cd9-49d3-95cf-1005c50eea22-kube-api-access-sxdzr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:50.930533 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.930474    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:50.930533 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:50.930484    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734b4d61-4cd9-49d3-95cf-1005c50eea22-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:51.614945 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:51.614858    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv"
Oct 30 23:10:51.614945 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:51.614890    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-45jpv" event={"ID":"734b4d61-4cd9-49d3-95cf-1005c50eea22","Type":"ContainerDied","Data":"e4f31cb64ff0ee52cebdf081e867f470e958510b0f42cc2af77b355cf8a66ccd"}
Oct 30 23:10:51.614945 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:10:51.614939    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f31cb64ff0ee52cebdf081e867f470e958510b0f42cc2af77b355cf8a66ccd"
Oct 30 23:10:58.437751 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:10:58.437709    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:11:09.845021 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:09.844952    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:10.010450 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.010412    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:10.202424 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.202347    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"]
Oct 30 23:11:10.202912 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.202892    5028 cpu_manager.go:414] "containerMap: removing container" podUID="734b4d61-4cd9-49d3-95cf-1005c50eea22" containerName="revoke-keys"
Oct 30 23:11:10.202992 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.202914    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b4d61-4cd9-49d3-95cf-1005c50eea22" containerName="revoke-keys"
Oct 30 23:11:10.203110 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.203031    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b4d61-4cd9-49d3-95cf-1005c50eea22" containerName="revoke-keys"
Oct 30 23:11:10.206843 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.206825    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.258001 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.257968    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:11:10.305255 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.305218    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:11:10.305255 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.305224    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:11:10.305517 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.305333    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d5msm\""
Oct 30 23:11:10.308006 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.307979    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"]
Oct 30 23:11:10.374274 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.374234    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24p6\" (UniqueName: \"kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.374469 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.374290    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.374469 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.374333    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.475889 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.475850    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24p6\" (UniqueName: \"kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.476114 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.475908    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.476114 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.475944    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.476636 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.476607    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.478417 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.478388    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.535046 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.535012    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24p6\" (UniqueName: \"kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6\") pod \"maas-api-revoke-sub-0165d7d2-lsgkz\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.824805 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.824717    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:10.877377 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.877348    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 23:11:10.913904 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:10.913877    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q2lps_e0d05eba-0c7d-4837-adc6-fa3fee9b4f2b/console-operator/2.log"
Oct 30 23:11:11.049539 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.049504    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"]
Oct 30 23:11:11.051911 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.051884    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cc4vn"]
Oct 30 23:11:11.202911 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.202882    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"]
Oct 30 23:11:11.204969 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:11:11.204940    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa9c626_5e5e_4fbd_9e33_8e7515e73bb3.slice/crio-c79ce82d0766a97bc4cedacde18f2942122968e13a35132b6b1335af1717e9ed WatchSource:0}: Error finding container c79ce82d0766a97bc4cedacde18f2942122968e13a35132b6b1335af1717e9ed: Status 404 returned error can't find the container with id c79ce82d0766a97bc4cedacde18f2942122968e13a35132b6b1335af1717e9ed
Oct 30 23:11:11.289909 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.289876    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a474e8d2-c5b4-44a6-a5f3-e4a2feb76173" path="/var/lib/kubelet/pods/a474e8d2-c5b4-44a6-a5f3-e4a2feb76173/volumes"
Oct 30 23:11:11.805182 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.805096    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz" event={"ID":"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3","Type":"ContainerStarted","Data":"a46a6d5ce1534b285abdef1d53236cf89714fc0c85d5efa7bbe5b9f3efe63cc4"}
Oct 30 23:11:11.805182 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.805151    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz" event={"ID":"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3","Type":"ContainerStarted","Data":"c79ce82d0766a97bc4cedacde18f2942122968e13a35132b6b1335af1717e9ed"}
Oct 30 23:11:11.917281 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:11.917228    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz" podStartSLOduration=1.9172154209999999 podStartE2EDuration="1.917215421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:11.916527765 +0000 UTC m=+4202.766040739" watchObservedRunningTime="2026-10-30 23:11:11.917215421 +0000 UTC m=+4202.766728385"
Oct 30 23:11:12.812798 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:12.812767    5028 generic.go:356] "Generic (PLEG): container finished" podID="3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" containerID="a46a6d5ce1534b285abdef1d53236cf89714fc0c85d5efa7bbe5b9f3efe63cc4" exitCode=0
Oct 30 23:11:12.813007 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:12.812828    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz" event={"ID":"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3","Type":"ContainerDied","Data":"a46a6d5ce1534b285abdef1d53236cf89714fc0c85d5efa7bbe5b9f3efe63cc4"}
Oct 30 23:11:13.976382 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:13.976352    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:14.112662 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.112581    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca\") pod \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") "
Oct 30 23:11:14.112662 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.112638    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6\" (UniqueName: \"kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6\") pod \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") "
Oct 30 23:11:14.112941 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.112745    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca\") pod \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\" (UID: \"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3\") "
Oct 30 23:11:14.113055 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.113026    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca" pod "3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" (UID: "3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:11:14.114958 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.114921    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca" pod "3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" (UID: "3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:11:14.115067 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.115006    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6" pod "3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" (UID: "3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3"). InnerVolumeSpecName "kube-api-access-r24p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:11:14.214325 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.214289    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:11:14.214325 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.214322    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:11:14.214538 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.214337    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24p6\" (UniqueName: \"kubernetes.io/projected/3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3-kube-api-access-r24p6\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:11:14.829066 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.829036    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz"
Oct 30 23:11:14.829257 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.829072    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lsgkz" event={"ID":"3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3","Type":"ContainerDied","Data":"c79ce82d0766a97bc4cedacde18f2942122968e13a35132b6b1335af1717e9ed"}
Oct 30 23:11:14.829257 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:14.829125    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79ce82d0766a97bc4cedacde18f2942122968e13a35132b6b1335af1717e9ed"
Oct 30 23:11:21.847883 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.847843    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"]
Oct 30 23:11:21.848394 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.848270    5028 cpu_manager.go:414] "containerMap: removing container" podUID="3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" containerName="revoke-keys"
Oct 30 23:11:21.848394 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.848282    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" containerName="revoke-keys"
Oct 30 23:11:21.848394 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.848383    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3" containerName="revoke-keys"
Oct 30 23:11:21.851759 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.851738    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.888201 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.888161    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.888368 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.888297    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.888368 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.888342    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd4gr\" (UniqueName: \"kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.950629 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.950594    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:11:21.950828 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.950721    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:11:21.950828 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.950766    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d5msm\""
Oct 30 23:11:21.950828 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.950786    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:11:21.953553 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.953535    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"]
Oct 30 23:11:21.989340 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.989303    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.989519 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.989361    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.989519 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.989393    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4gr\" (UniqueName: \"kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.990039 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.990007    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:21.991890 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:21.991864    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:22.048532 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:22.048500    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd4gr\" (UniqueName: \"kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr\") pod \"maas-api-revoke-sub-6b19338f-6chd6\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:22.165209 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:22.165116    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:22.533656 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:22.533624    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"]
Oct 30 23:11:22.535157 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:11:22.535122    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448555a6_d05b_462f_bf12_1ae1eaf95fd5.slice/crio-7ad48ff75940ba1f80457dcab170790640d19c042be489ce857b74d2635dde71 WatchSource:0}: Error finding container 7ad48ff75940ba1f80457dcab170790640d19c042be489ce857b74d2635dde71: Status 404 returned error can't find the container with id 7ad48ff75940ba1f80457dcab170790640d19c042be489ce857b74d2635dde71
Oct 30 23:11:22.881670 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:22.881631    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6" event={"ID":"448555a6-d05b-462f-bf12-1ae1eaf95fd5","Type":"ContainerStarted","Data":"7869629780dbff7daa5809a43472f4a342b61ebea4756b0550f7fb8f8cd16494"}
Oct 30 23:11:22.882187 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:22.881729    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6" event={"ID":"448555a6-d05b-462f-bf12-1ae1eaf95fd5","Type":"ContainerStarted","Data":"7ad48ff75940ba1f80457dcab170790640d19c042be489ce857b74d2635dde71"}
Oct 30 23:11:22.992585 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:22.992531    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6" podStartSLOduration=1.992516604 podStartE2EDuration="1.992516604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:22.99142483 +0000 UTC m=+4213.840937805" watchObservedRunningTime="2026-10-30 23:11:22.992516604 +0000 UTC m=+4213.842029576"
Oct 30 23:11:23.889655 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:23.889620    5028 generic.go:356] "Generic (PLEG): container finished" podID="448555a6-d05b-462f-bf12-1ae1eaf95fd5" containerID="7869629780dbff7daa5809a43472f4a342b61ebea4756b0550f7fb8f8cd16494" exitCode=0
Oct 30 23:11:23.890098 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:23.889737    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6" event={"ID":"448555a6-d05b-462f-bf12-1ae1eaf95fd5","Type":"ContainerDied","Data":"7869629780dbff7daa5809a43472f4a342b61ebea4756b0550f7fb8f8cd16494"}
Oct 30 23:11:24.050919 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:24.050887    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:11:25.041938 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.041908    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:25.122909 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.122866    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca\") pod \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") "
Oct 30 23:11:25.122909 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.122917    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr\" (UniqueName: \"kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr\") pod \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") "
Oct 30 23:11:25.123131 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.122953    5028 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca\") pod \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\" (UID: \"448555a6-d05b-462f-bf12-1ae1eaf95fd5\") "
Oct 30 23:11:25.123387 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.123363    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca" pod "448555a6-d05b-462f-bf12-1ae1eaf95fd5" (UID: "448555a6-d05b-462f-bf12-1ae1eaf95fd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:11:25.125058 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.125028    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca" pod "448555a6-d05b-462f-bf12-1ae1eaf95fd5" (UID: "448555a6-d05b-462f-bf12-1ae1eaf95fd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:11:25.125161 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.125107    5028 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr" pod "448555a6-d05b-462f-bf12-1ae1eaf95fd5" (UID: "448555a6-d05b-462f-bf12-1ae1eaf95fd5"). InnerVolumeSpecName "kube-api-access-pd4gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:11:25.224301 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.224268    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-openshift-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:11:25.224301 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.224299    5028 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448555a6-d05b-462f-bf12-1ae1eaf95fd5-maas-api-xks-ca\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:11:25.224301 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.224309    5028 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd4gr\" (UniqueName: \"kubernetes.io/projected/448555a6-d05b-462f-bf12-1ae1eaf95fd5-kube-api-access-pd4gr\") on node \"ip-10-0-129-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:11:25.903079 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.903048    5028 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6"
Oct 30 23:11:25.903270 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.903054    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6chd6" event={"ID":"448555a6-d05b-462f-bf12-1ae1eaf95fd5","Type":"ContainerDied","Data":"7ad48ff75940ba1f80457dcab170790640d19c042be489ce857b74d2635dde71"}
Oct 30 23:11:25.903270 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:25.903174    5028 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad48ff75940ba1f80457dcab170790640d19c042be489ce857b74d2635dde71"
Oct 30 23:11:28.436053 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:11:28.436010    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:11:34.005044 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:34.004981    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58989f478d-zkntj_107baba0-bcf3-4809-9aab-fdbe79fdfa80/authorino/0.log"
Oct 30 23:11:36.431892 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:36.431849    5028 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rwbtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:11:36.432585 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:36.432535    5028 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rwbtf" podUID="a143fab9-aead-4204-80e5-c8a690c95513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:11:44.048332 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:44.048292    5028 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"]
Oct 30 23:11:44.095807 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:44.095773    5028 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e8d181-n9fkr"]
Oct 30 23:11:45.102357 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:45.102327    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7cd4b5bfc5-nqqbs_8eed38c4-f9c7-4635-925c-0b5cc8e51ebd/manager/0.log"
Oct 30 23:11:45.289911 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:45.289869    5028 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7779d432-9922-4090-8771-a2d602cac79f" path="/var/lib/kubelet/pods/7779d432-9922-4090-8771-a2d602cac79f/volumes"
Oct 30 23:11:49.225993 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:49.225964    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-lsgkz_3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3/revoke-keys/0.log"
Oct 30 23:11:49.455105 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:49.455075    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-6chd6_448555a6-d05b-462f-bf12-1ae1eaf95fd5/revoke-keys/0.log"
Oct 30 23:11:49.681731 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:49.681634    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-45jpv_734b4d61-4cd9-49d3-95cf-1005c50eea22/revoke-keys/0.log"
Oct 30 23:11:51.480518 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:51.480488    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58989f478d-zkntj_107baba0-bcf3-4809-9aab-fdbe79fdfa80/authorino/0.log"
Oct 30 23:11:51.961385 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:51.961354    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rcnrs_4c8e1dae-6c09-40ea-929a-17f1e515a6f2/manager/0.log"
Oct 30 23:11:52.210794 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:52.210754    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zs6bt_779e849b-6f55-4500-9b81-aba988308aba/manager/0.log"
Oct 30 23:11:52.462509 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:52.462486    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lbp5n_a17310a7-9513-449e-a500-ffb813321d7b/kuadrant-console-plugin/0.log"
Oct 30 23:11:52.766289 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:52.766207    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6rk9z_cb45a923-6e92-4af4-b616-7309a32c021e/manager/0.log"
Oct 30 23:11:53.320588 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:53.320555    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xcz7l_274237e7-b77d-4d95-92ab-a3cb728bf913/manager/0.log"
Oct 30 23:11:54.154662 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:54.154629    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-th9dm_d409403f-fbdb-4f41-bc4f-9f41afc66d88/istio-proxy/0.log"
Oct 30 23:11:55.366548 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:55.366508    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-5c66s_f1f27ceb-9be0-407c-a559-b5e3338ad1d3/istio-proxy/0.log"
Oct 30 23:11:55.839733 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:55.839703    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-dgkjv_7b61aded-a35f-42a1-8753-462f487ccfef/istio-proxy/0.log"
Oct 30 23:11:57.169348 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:57.169318    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rwbtf_a143fab9-aead-4204-80e5-c8a690c95513/istio-proxy/0.log"
Oct 30 23:11:57.463310 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:57.463278    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-wgcfm_b3fe4656-786a-40d9-977c-83d2bbf357cd/istio-proxy/0.log"
Oct 30 23:11:57.750941 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:57.750868    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk_e5cca595-c64b-4d4c-a9be-b955ec62c585/istio-proxy/0.log"
Oct 30 23:11:57.986930 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:57.986903    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t9lxh_739abe96-0923-4b32-8e9b-c4e66f7237da/payload-pre-processing/0.log"
Oct 30 23:11:58.217045 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:58.217016    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fdzns_7c08f8f7-8f06-4532-8c79-43d5d77ce9dc/payload-processing/0.log"
Oct 30 23:11:58.435577 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:11:58.435539    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:11:58.446153 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:11:58.446124    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b69957bf-6kffb_56a790e5-683e-4dfd-82e1-1bde4629e09f/router/0.log"
Oct 30 23:12:00.773891 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:00.773855    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t_9b57c28a-0d5d-4eac-bee0-7f18f78aac56/main/0.log"
Oct 30 23:12:01.010190 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:01.010156    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk_5d7eba18-77c0-4727-8a41-a77ee94779b5/main/0.log"
Oct 30 23:12:01.978168 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:01.978138    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58989f478d-zkntj_107baba0-bcf3-4809-9aab-fdbe79fdfa80/authorino/0.log"
Oct 30 23:12:11.571678 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:11.571644    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7cd4b5bfc5-nqqbs_8eed38c4-f9c7-4635-925c-0b5cc8e51ebd/manager/0.log"
Oct 30 23:12:15.018523 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:15.018490    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-lsgkz_3aa9c626-5e5e-4fbd-9e33-8e7515e73bb3/revoke-keys/0.log"
Oct 30 23:12:15.268632 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:15.268545    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-6chd6_448555a6-d05b-462f-bf12-1ae1eaf95fd5/revoke-keys/0.log"
Oct 30 23:12:15.502878 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:15.502843    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-45jpv_734b4d61-4cd9-49d3-95cf-1005c50eea22/revoke-keys/0.log"
Oct 30 23:12:16.953333 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:16.953299    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58989f478d-zkntj_107baba0-bcf3-4809-9aab-fdbe79fdfa80/authorino/0.log"
Oct 30 23:12:17.436157 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:17.436123    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rcnrs_4c8e1dae-6c09-40ea-929a-17f1e515a6f2/manager/0.log"
Oct 30 23:12:17.685282 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:17.685248    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zs6bt_779e849b-6f55-4500-9b81-aba988308aba/manager/0.log"
Oct 30 23:12:17.924041 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:17.924012    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lbp5n_a17310a7-9513-449e-a500-ffb813321d7b/kuadrant-console-plugin/0.log"
Oct 30 23:12:18.166670 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:18.166637    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6rk9z_cb45a923-6e92-4af4-b616-7309a32c021e/manager/0.log"
Oct 30 23:12:18.688393 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:18.688363    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xcz7l_274237e7-b77d-4d95-92ab-a3cb728bf913/manager/0.log"
Oct 30 23:12:19.333033 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:19.332946    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-th9dm_d409403f-fbdb-4f41-bc4f-9f41afc66d88/istio-proxy/0.log"
Oct 30 23:12:20.167717 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:20.167652    5028 scope.go:122] "RemoveContainer" containerID="84a0b5dafdc48a8016c4b2393df7c8c360a823bb9f03612920244094e11d01ca"
Oct 30 23:12:20.179000 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:20.178979    5028 scope.go:122] "RemoveContainer" containerID="b45ca9dc3cd0fe85f7376c7df6108c115b02b9a4f1de448a8fd24b0ba701b1d9"
Oct 30 23:12:20.534966 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:20.534937    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-5c66s_f1f27ceb-9be0-407c-a559-b5e3338ad1d3/istio-proxy/0.log"
Oct 30 23:12:21.001972 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:21.001944    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-dgkjv_7b61aded-a35f-42a1-8753-462f487ccfef/istio-proxy/0.log"
Oct 30 23:12:22.320234 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:22.320201    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rwbtf_a143fab9-aead-4204-80e5-c8a690c95513/istio-proxy/0.log"
Oct 30 23:12:22.561797 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:22.561770    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-wgcfm_b3fe4656-786a-40d9-977c-83d2bbf357cd/istio-proxy/0.log"
Oct 30 23:12:22.845387 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:22.845355    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-q4pbk_e5cca595-c64b-4d4c-a9be-b955ec62c585/istio-proxy/0.log"
Oct 30 23:12:23.082581 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:23.082546    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-t9lxh_739abe96-0923-4b32-8e9b-c4e66f7237da/payload-pre-processing/0.log"
Oct 30 23:12:23.327650 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:23.327622    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fdzns_7c08f8f7-8f06-4532-8c79-43d5d77ce9dc/payload-processing/0.log"
Oct 30 23:12:23.580180 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:23.580085    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b69957bf-6kffb_56a790e5-683e-4dfd-82e1-1bde4629e09f/router/0.log"
Oct 30 23:12:24.429740 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:24.429676    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:25.609447 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:25.609397    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8x42t_9b57c28a-0d5d-4eac-bee0-7f18f78aac56/main/0.log"
Oct 30 23:12:25.849645 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:25.849614    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nwllk_5d7eba18-77c0-4727-8a41-a77ee94779b5/main/0.log"
Oct 30 23:12:27.730364 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:27.730329    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:27.884682 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:27.884645    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:12:28.435826 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:12:28.435789    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:12:56.807614 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:56.807588    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fdzns_7c08f8f7-8f06-4532-8c79-43d5d77ce9dc/payload-processing/0.log"
Oct 30 23:12:57.045758 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:12:57.045724    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fdzns_7c08f8f7-8f06-4532-8c79-43d5d77ce9dc/payload-processing/0.log"
Oct 30 23:12:58.435094 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:12:58.435058    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:13:20.209911 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:20.209880    5028 scope.go:122] "RemoveContainer" containerID="5a24eb3145fc2d527732ac011cb1f4b5735c6ca2d1949499417cb74c2d69d2d7"
Oct 30 23:13:24.786478 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:24.786440    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:28.434838 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:13:28.434800    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:13:33.195042 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:33.194998    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:33.343647 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:33.343600    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:36.462307 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:36.462281    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fdzns_7c08f8f7-8f06-4532-8c79-43d5d77ce9dc/payload-processing/0.log"
Oct 30 23:13:36.684256 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:36.684226    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fdzns_7c08f8f7-8f06-4532-8c79-43d5d77ce9dc/payload-processing/0.log"
Oct 30 23:13:37.047202 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.047165    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2xpx/must-gather-zhzf4"]
Oct 30 23:13:37.047716 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.047675    5028 cpu_manager.go:414] "containerMap: removing container" podUID="448555a6-d05b-462f-bf12-1ae1eaf95fd5" containerName="revoke-keys"
Oct 30 23:13:37.047716 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.047703    5028 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448555a6-d05b-462f-bf12-1ae1eaf95fd5" containerName="revoke-keys"
Oct 30 23:13:37.047823 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.047774    5028 memory_manager.go:365] "RemoveStaleState removing state" podUID="448555a6-d05b-462f-bf12-1ae1eaf95fd5" containerName="revoke-keys"
Oct 30 23:13:37.052444 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.052424    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.053403 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.053376    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:37.102396 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.102367    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2xpx\"/\"kube-root-ca.crt\""
Oct 30 23:13:37.113505 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.113473    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43b62f87-4de4-40f0-a87a-4ee89b1b07d2-must-gather-output\") pod \"must-gather-zhzf4\" (UID: \"43b62f87-4de4-40f0-a87a-4ee89b1b07d2\") " pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.113665 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.113635    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktxk\" (UniqueName: \"kubernetes.io/projected/43b62f87-4de4-40f0-a87a-4ee89b1b07d2-kube-api-access-6ktxk\") pod \"must-gather-zhzf4\" (UID: \"43b62f87-4de4-40f0-a87a-4ee89b1b07d2\") " pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.149600 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.149564    5028 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2xpx\"/\"default-dockercfg-xvgbt\""
Oct 30 23:13:37.149600 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.149598    5028 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2xpx\"/\"openshift-service-ca.crt\""
Oct 30 23:13:37.152359 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.152335    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/must-gather-zhzf4"]
Oct 30 23:13:37.214651 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.214616    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43b62f87-4de4-40f0-a87a-4ee89b1b07d2-must-gather-output\") pod \"must-gather-zhzf4\" (UID: \"43b62f87-4de4-40f0-a87a-4ee89b1b07d2\") " pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.214902 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.214751    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktxk\" (UniqueName: \"kubernetes.io/projected/43b62f87-4de4-40f0-a87a-4ee89b1b07d2-kube-api-access-6ktxk\") pod \"must-gather-zhzf4\" (UID: \"43b62f87-4de4-40f0-a87a-4ee89b1b07d2\") " pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.215079 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.215057    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43b62f87-4de4-40f0-a87a-4ee89b1b07d2-must-gather-output\") pod \"must-gather-zhzf4\" (UID: \"43b62f87-4de4-40f0-a87a-4ee89b1b07d2\") " pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.226974 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.226923    5028 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log" limit=1024
Oct 30 23:13:37.273259 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.273231    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktxk\" (UniqueName: \"kubernetes.io/projected/43b62f87-4de4-40f0-a87a-4ee89b1b07d2-kube-api-access-6ktxk\") pod \"must-gather-zhzf4\" (UID: \"43b62f87-4de4-40f0-a87a-4ee89b1b07d2\") " pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.367160 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.367071    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/must-gather-zhzf4"
Oct 30 23:13:37.736904 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.736874    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/must-gather-zhzf4"]
Oct 30 23:13:37.738203 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:13:37.738178    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43b62f87_4de4_40f0_a87a_4ee89b1b07d2.slice/crio-0b065d3335c5a02530db28c86a41770182dc66a4597e43de12fa6459f31c70c4 WatchSource:0}: Error finding container 0b065d3335c5a02530db28c86a41770182dc66a4597e43de12fa6459f31c70c4: Status 404 returned error can't find the container with id 0b065d3335c5a02530db28c86a41770182dc66a4597e43de12fa6459f31c70c4
Oct 30 23:13:37.740400 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:37.740382    5028 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:13:38.710665 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:38.710626    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/must-gather-zhzf4" event={"ID":"43b62f87-4de4-40f0-a87a-4ee89b1b07d2","Type":"ContainerStarted","Data":"0b065d3335c5a02530db28c86a41770182dc66a4597e43de12fa6459f31c70c4"}
Oct 30 23:13:39.718573 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:39.718532    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/must-gather-zhzf4" event={"ID":"43b62f87-4de4-40f0-a87a-4ee89b1b07d2","Type":"ContainerStarted","Data":"b02f9d40294778ea3f55abfd86dfb785fcdb0fe38131dfeba89ee54c45cfbb33"}
Oct 30 23:13:39.719022 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:39.718583    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/must-gather-zhzf4" event={"ID":"43b62f87-4de4-40f0-a87a-4ee89b1b07d2","Type":"ContainerStarted","Data":"9a66c6ad4bd9fc43cff9ed4ece2392ad0b2be3cb6c1a16645e4f3c03e487e4fa"}
Oct 30 23:13:39.830037 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:39.829981    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q2xpx/must-gather-zhzf4" podStartSLOduration=1.474469268 podStartE2EDuration="2.829964005s" totalImagesPullingTime="1.355494737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:39.828450273 +0000 UTC m=+4350.677963250" watchObservedRunningTime="2026-10-30 23:13:39.829964005 +0000 UTC m=+4350.679476991"
Oct 30 23:13:47.027706 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:47.027373    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rbvwt_8ebd318a-c0d2-45e5-8eeb-f5d85fa0b165/global-pull-secret-syncer/0.log"
Oct 30 23:13:47.657712 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:47.654413    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7dn4w_03d17bd7-a7c6-45b9-b0e5-2d9809543781/connection-checker/0.log"
Oct 30 23:13:48.075981 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:48.075844    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cgm7l_1a535db2-2694-4965-b4f9-5b478681381e/connection-checker/0.log"
Oct 30 23:13:48.504579 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:48.504528    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-csjz8_5240e9dc-63a3-436f-af94-ab88f6af0c36/connection-checker/0.log"
Oct 30 23:13:49.005707 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:49.005068    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4j6gf_d28a3c9a-7726-406c-bfdf-f36065f02a81/konnectivity-agent/0.log"
Oct 30 23:13:50.302489 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:50.302458    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-125.us-west-2.compute.internal_95227d9216a597ca813ef85310eb7d77/haproxy/0.log"
Oct 30 23:13:56.174757 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.174673    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58989f478d-zkntj_107baba0-bcf3-4809-9aab-fdbe79fdfa80/authorino/0.log"
Oct 30 23:13:56.502032 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.501998    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rcnrs_4c8e1dae-6c09-40ea-929a-17f1e515a6f2/manager/0.log"
Oct 30 23:13:56.517903 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.517867    5028 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"]
Oct 30 23:13:56.525982 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.525954    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.579287 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.579168    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"]
Oct 30 23:13:56.603501 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.603465    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-proc\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.603703 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.603532    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnb8\" (UniqueName: \"kubernetes.io/projected/d9a7f017-6dcb-4a25-a739-82dec7fe356f-kube-api-access-sjnb8\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.603703 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.603615    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-podres\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.603703 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.603645    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-sys\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.603864 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.603772    5028 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-lib-modules\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705168 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705125    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-proc\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705179    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnb8\" (UniqueName: \"kubernetes.io/projected/d9a7f017-6dcb-4a25-a739-82dec7fe356f-kube-api-access-sjnb8\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705203    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-podres\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705218    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-sys\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705269    5028 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-lib-modules\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705274    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-proc\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705359    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-sys\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705367    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-lib-modules\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.705380 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.705358    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9a7f017-6dcb-4a25-a739-82dec7fe356f-podres\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.764815 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.764733    5028 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnb8\" (UniqueName: \"kubernetes.io/projected/d9a7f017-6dcb-4a25-a739-82dec7fe356f-kube-api-access-sjnb8\") pod \"perf-node-gather-daemonset-pqrbr\" (UID: \"d9a7f017-6dcb-4a25-a739-82dec7fe356f\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.838354 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.838323    5028 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:56.938902 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:56.938871    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zs6bt_779e849b-6f55-4500-9b81-aba988308aba/manager/0.log"
Oct 30 23:13:57.215962 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.215932    5028 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"]
Oct 30 23:13:57.218161 ip-10-0-129-125 kubenswrapper[5028]: W1030 23:13:57.218131    5028 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9a7f017_6dcb_4a25_a739_82dec7fe356f.slice/crio-b3d9bc48489145312f3abf575c8098d7a97b438cc55ee19ec1e394fccbca39aa WatchSource:0}: Error finding container b3d9bc48489145312f3abf575c8098d7a97b438cc55ee19ec1e394fccbca39aa: Status 404 returned error can't find the container with id b3d9bc48489145312f3abf575c8098d7a97b438cc55ee19ec1e394fccbca39aa
Oct 30 23:13:57.370596 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.370559    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lbp5n_a17310a7-9513-449e-a500-ffb813321d7b/kuadrant-console-plugin/0.log"
Oct 30 23:13:57.836901 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.836865    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6rk9z_cb45a923-6e92-4af4-b616-7309a32c021e/manager/0.log"
Oct 30 23:13:57.876014 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.875974    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr" event={"ID":"d9a7f017-6dcb-4a25-a739-82dec7fe356f","Type":"ContainerStarted","Data":"4920873e7c0187369ab4dd733234e5b94de42a2e262c097c1a1f337e8ddd62a8"}
Oct 30 23:13:57.876209 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.876032    5028 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr" event={"ID":"d9a7f017-6dcb-4a25-a739-82dec7fe356f","Type":"ContainerStarted","Data":"b3d9bc48489145312f3abf575c8098d7a97b438cc55ee19ec1e394fccbca39aa"}
Oct 30 23:13:57.876209 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.876195    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:13:57.988759 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:57.988682    5028 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr" podStartSLOduration=1.988668101 podStartE2EDuration="1.988668101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:57.98738166 +0000 UTC m=+4368.836894633" watchObservedRunningTime="2026-10-30 23:13:57.988668101 +0000 UTC m=+4368.838181074"
Oct 30 23:13:58.436680 ip-10-0-129-125 kubenswrapper[5028]: E1030 23:13:58.436638    5028 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nxvq5" containerName="extractor"
Oct 30 23:13:58.692939 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:13:58.692826    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xcz7l_274237e7-b77d-4d95-92ab-a3cb728bf913/manager/0.log"
Oct 30 23:14:03.895512 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:14:03.895487    5028 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-pqrbr"
Oct 30 23:14:04.322008 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:14:04.321971    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2hrpg_73ea9998-7cb8-45d2-ad2b-473bd8f4ae7e/cluster-monitoring-operator/0.log"
Oct 30 23:14:09.515950 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:14:09.515918    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tcd2z_10a9c7f4-272c-472a-acd1-b76ff3e29846/node-exporter/0.log"
Oct 30 23:14:09.942920 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:14:09.942842    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tcd2z_10a9c7f4-272c-472a-acd1-b76ff3e29846/kube-rbac-proxy/0.log"
Oct 30 23:14:10.369018 ip-10-0-129-125 kubenswrapper[5028]: I1030 23:14:10.368991    5028 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tcd2z_10a9c7f4-272c-472a-acd1-b76ff3e29846/init-textfile/0.log"